How do you concurrently execute long running tasks that process a high volume of data or require complex processing? To achieve maximum throughput, you need to process these tasks as a series of cooperating, dependent stages, i.e. you need to pipeline your tasks. This session describes how to design and manage a thread pipeline that achieves maximum throughput, while avoiding the common pitfalls of cooperating threads: blocking, hanging, deadlock, etc. The intended audience is Java technology developers who want to increase their application performance. Key points: • When and how to pipeline • Starting and stopping pipeline stages • Stage cooperation and communication • Bulletproof handling for thread failures

1. Phillip Koza
IBM
Maximizing Throughput and
Reliability with Pipelined Tasks
JavaOneJavaOne

2. JavaOneJavaOne
2
Agenda
What is pipelining?
When to pipeline
Creating/starting stages and threads
Stage/thread communication
Waiting/flushing
Stopping stages/threads
Uncaught exception handling
Shutdown hooks, Daemon threads
Summary

3. JavaOneJavaOne
3
What is pipelining?

4. JavaOneJavaOne
4

5. JavaOneJavaOne
5
What is pipelining?
Pipelining is processing a given job as a series of
cooperating, dependent, sequential stages
Each stage does a subset of the processing, then hands off
to the next stage
Each stage is a separate thread or threads
Each stage has at least one input queue and at least one
output queue
These queues may be logical
Don’t want a stage to get too far ahead of its successor
Stages should block if they are too far ahead

6. JavaOneJavaOne
6
What is pipelining?
Well understood and utilized concept in microprocessors:
almost all processor chips are heavily pipelined
 Pipelining is operator-based parallelism, as opposed to data
(partition-based) parallelism
 Each stage executes in parallel a subset of the operations
that need to be performed on all objects/tasks
 Partition based parallelism executes in parallel all
operations on a subset of the objects/tasks
Both forms of parallelism can be utilized at once!

7. JavaOneJavaOne
7
Example – Stock data app
StoreStore
ParseParse
Read allRead all
stockstock
datadata
TechnologyTechnology
stockstock
data subscriberdata subscriber
CommodityCommodity
stockstock
data subscriberdata subscriber
Financial stockFinancial stock
data subscriberdata subscriber

8. JavaOneJavaOne
8
Why pipelining?
Increased throughput , responsiveness, and processor
utilization
If n stages, a well balanced pipeline can increase cpu
utilization and throughput by a factor of close to n
Other stages can continue processing if a stage must perform
I/O
Flexibility: easy to add additional processing – just add
another stage!
Maintainability: code follows the pipeline – harder to have
monolithic 3000 line methods that do everything!

9. JavaOneJavaOne
9
Pipeline topologies
Basic: Single producer to single consumer
Pipelining + partitioned input processing: multiple
producers to single consumer
Each producer processes a partition of the input
Pipelining + partitioned output processing: single producer
to multiple consumers
Each consumer processes a partition of the output
Multiple producers + multiple consumers
Complicated: if possible, have separate pipelines!

10. JavaOneJavaOne
10
When to pipeline
Complex job with multiple processing modules
Stage candidates: when should you consider having a
module run as a separate stage?
Running in a separate thread is not free: must consider
additional startup, synchronization, complexity
The benefits must be weighed against these additional costs
Luckily, there are ways to minimize these costs!
A key to effective pipelining is determining which modules
to run as pipeline stages

11. JavaOneJavaOne
11
When to pipeline
Module does I/O
Module does significant processing that is not interleaved
with that of other stages
Profiling shows this module is a bottleneck

12. JavaOneJavaOne
12
When to pipeline
Ideally, all stages are perfectly balanced: if n stages, each
stage does 1/n of the work
Each stage then never waits for input (except at startup) and
never waits to deliver output
If not perfectly balanced, some stages may get ahead of the
others
These stages should block until the other stages catch up
Number of stages is usually statically fixed when the
pipeline starts

13. JavaOneJavaOne
13
Partition parallelism
 Partition parallelism is having separate threads process a portion
(partition) of the data stream
 The partitions are not necessarily disjoint
 Can be utilized whenever objects can be processed in any order,
OR you know how to merge them into the correct order
 Can be used to increase overall throughput, or throughput of
individual stages
 If objects can be processed in any order, can have an independent
pipeline for each partition
 Overhead may be too high if some stages are very lightweight
 If dependencies, must merge the streams at some point – so need a
common pipeline
 Partition threads can be allocated statically or dynamically

14. JavaOneJavaOne
14
How many threads?
Keep track of how often consumers wait for producers and
vice-versa
Lots of consumer waits = more producer threads
Lots of producer waits = more consumer threads
This can mean adding more pipelines, more pipeline stages,
or more threads per stage
Adding more threads to slow stages is particularly useful for
balancing stages

15. JavaOneJavaOne
15
When to pipeline
Example: Determine stage boundaries
To compile a SQL query, the SQL compiler must:
Lexically analyze the query
Parse the token stream from the lexer
Bind objects (table, column, etc)
Normalize
Optimize
Generate query plan
Currently one thread is doing all of the above processing

16. JavaOneJavaOne
16
When to pipeline
Example: Determine stage boundaries
Analysis shows that:
Lexer, Parser and Binder use significant CPU and can do I/O
(catalog table lookups)
 But calls to these modules can be interleaved, so should not be split
 Binding a view will require the underlying query expression to be lexically
analyzed and parsed
Normalizer, optimizer, and code generator all use significant
CPU and these calls are not interleaved

17. JavaOneJavaOne
17
When to pipeline
Example: Determine stage boundaries
Split into the following stages:
Lexer/Parser/Binder
Normalizer
Optimizer
Code generator
 Each stage can then execute in parallel
Normalization, optimization, and code generation can continue
if the Binder must do I/O
Each stage of processing proceeds in parallel

18. JavaOneJavaOne
18
Creating threads
ExecutorService or raw threads?

19. JavaOneJavaOne
19
Creating threads
ThreadPoolExecutor
Implements ExecutorService
Best when have lots of independent short running tasks that
terminate themselves
Provides automatic thread reuse via pooling
Pooling pays off big for short running tasks

20. JavaOneJavaOne
20
Creating threads
ThreadPoolExecutor
Creates threads via ThreadFactory
Default is Executors.defaultThreadFactory
Can create your own and set via setThreadFactory
Can specify own thread name, priority, group, etc

21. JavaOneJavaOne
21
Creating threads
 ThreadPoolExecutor
 In a pipeline, this can be useful for creating additional threads for
partitioned parallelism for a pipeline stage
 Each stage has its own ThreadPoolExecutor
 Threads can be added as the stage load increases, returned to the
pool when the stage load decreases
 Control maximum number of idle threads and total threads by
setting corePoolSize and maximumPoolSize
 It’s important to limit the maximumPoolSize
 Unlimited growth in the number of threads can quickly exhaust all
resources

22. JavaOneJavaOne
22
Creating threads
ThreadPoolExecutor
Unless an unlimited maximumPoolSize is specified, a
request for another thread will block waiting for a thread to
become available
If no more threads immediately available, would rather just
use one of the existing threads
So use a SynchronousQueue as the work queue
Using the default RejectedExecutionHandler, the request will
be rejected if the maximum number of threads are already in
use
 i.e. a RejectedExecutionException will be thrown

23. JavaOneJavaOne
23
Using ThreadPoolExecutor
private static final maxThreads = 5;
private SynchronousQueue syncQ = new SynchronousQueue();
private ExecutorService threadPool = new ThreadPoolExecutor(maxThreads,
maxThreads, 60, TimeUnit.SECONDS,syncQ);
public boolean addWorkerThread(StageWorker worker) {
try {
threadPool.execute(new StageRunnable(worker));
}
catch (RejectedExecutionException) {
return false;
}
return true;
}

24. JavaOneJavaOne
24
Creating threads
Raw threads
Gives maximum control over when threads are created,
started, and stopped
The best choice when you have long running dependent
tasks, i.e. a pipeline
Pipelines need maximum control over their threads
All stage threads should be up, or all should be down
Need precise control over the timing of starting and stopping
threads

25. JavaOneJavaOne
25
Creating threads
 Pooling potentially useful, but not as important
 Thread creation is not the dominant cost
 Be very careful if have one giant thread pool for all stages and
pipelines!
 A pipeline does not function unless all stages are executing
 Don’t want some pipeline stages to be queued awaiting execution
while others are running
 If need to limit the total number of threads, it is best to limit the number
of pipelines
 For a pipeline, raw threads are the best choice – for now, at least!

26. JavaOneJavaOne
26
Using raw threads
private static final maxThreads = 5;
private threadCount = 0;
private final List<Thread> threadList = new ArrayList<Thread>(maxThreads);
public boolean addWorkerThread(StageWorker worker) {
if (threadCount < maxThreads) {
Thread stageThread = new Thread(new StageRunnable(worker));
threadList.add(stageThread);
threadCount++;
stageThread.start();
return true;
}
return false;
}

27. JavaOneJavaOne
27
Starting threads
Two basic control models:
MVC (Model/View/Controller)
Waterfall (hierarchical)

28. JavaOneJavaOne
28
Starting threads - MVC
Model is the data flow
View is presentation of the monitoring data/return data
Controller logic is kept separate
Dedicated controller thread
Controller must be aware of all stages/threads, how to start
them, and how they are linked
Controller can precisely control which threads get created
first

29. JavaOneJavaOne
29
Starting threads - MVC
Controller starts all stages/threads when the pipeline starts
Messages/callbacks to the controller are used to restart or
add additional stages/threads
Controller must ensure predecessor and successor stages
properly handle this
Controller can monitor pipeline state
If a stage thread dies unexpectedly, can restart the stage
But safest and easiest thing to do is to bring down the entire
pipeline

30. JavaOneJavaOne
30
MVC Diagram
Stage 3Stage 3
Sink
Source
ControllerController
Stage 2Stage 2
Stage 1Stage 1

31. JavaOneJavaOne
31
Starting threads - Waterfall
No dedicated control thread
Start from the head of the pipeline, i.e. the final consumer
stage/thread
Each successor (parent) stage creates any immediate
predecessor (child) stage threads that it needs
Control “falls” from parent to child, hence the name
If any child stage/thread dies, parent must notice and handle
(restart child, stop itself, etc)

32. JavaOneJavaOne
32
Starting threads - Waterfall
In effect, each parent thread acts as the controller for the
child
If additional child threads for a stage are needed, the parent
creates them
 Siblings should not create additional siblings – only a parent should!
(the parent may be quite happy with the number of children it
already has!)

33. JavaOneJavaOne
33
Waterfall Diagram
Stage 3Stage 3
Sink
Source
Stage 2Stage 2
Stage 1Stage 1
ChildChild
ParentParent
ChildChild
ParentParent
createscreates
createscreates

34. JavaOneJavaOne
34
Starting threads
MVC advantages
Components do not need to know who their predecessors or
successors are
All pipeline control logic is in one place
 Easy to refactor code to add/remove stages
 Makes maintenance easier
MVC disadvantages
Must be careful to avoid starting a predecessor stage before
the successor is ready

35. JavaOneJavaOne
35
Starting threads
Waterfall advantages
Parent has complete control of its child
 Can start and stop child threads as it pleases
Parent can easily guarantee that the child is not started until
the parent is ready
Waterfall disadvantages
Logic to start/stop stages is in each successor stage
 Harder to add/remove stages
 Each stage has its predecessor hard-coded
 Logic can become subtly different over time
 Separate logic harder to find, maintain

36. JavaOneJavaOne
36
Stage/thread communication
 How do you move data from one stage to the next?
 Single producer to single consumer
 Producer adds data to [logical] output queue, consumer reads
 Multiple producers to single consumer
 Each producer adds data to its [logical] output queue, consumer reads
from each producer queue, merging output if necessary
 Single producer to multiple consumers, consumers process any
available data
 Producer adds data to a single [logical] output queue, each consumer
reads next available

37. JavaOneJavaOne
37
Stage/thread communication
 Single producer to multiple consumers, consumers process a
partition of the data
 Each consumer requires a not necessarily distinct subset of the data
 Two potential models: push vs. pull
 Push: producer determines what data to give to each consumer,
and then sends it to each
 Producer has to know what the consumer wants
 Producer must allocate time to do this
 Producer may have data to give, but consumer is not ready
 Consumer may be ready for data, and the producer may have data to
give, but the producer is busy doing other things

38. JavaOneJavaOne
38
Stage/thread communication
Pull: Consumer gets the data it needs directly from the
producer
Producer doesn’t need to spend time determining what data a
consumer needs and sending it
If data is available, consumer doesn’t have to wait for the
producer to send it
 Requires consumer to access producer data structures –
which means synchronization!
Pull has the potential to perform better, but is more
complicated

39. JavaOneJavaOne
39
Push Example – Stock data app
Store,Store,
DistributeDistribute
ParseParse
Read allRead all
stockstock
datadata
queuequeue
queuequeue
queuequeue
TechnologyTechnology
stockstock
data subscriberdata subscriber
CommodityCommodity
stockstock
data subscriberdata subscriber
Financial stockFinancial stock
data subscriberdata subscriber

40. JavaOneJavaOne
40
Pull Example – Stock data app
StoreStoreParseParse
Read allRead all
stockstock
datadata
SynchSynch
AccessAccess
to storeto store
SynchSynch
AccessAccess
to storeto store
SynchSynch
AccessAccess
to storeto store
TechnologyTechnology
stockstock
data subscriberdata subscriber
Financial stockFinancial stock
data subscriberdata subscriber
CommodityCommodity
stockstock
data subscriberdata subscriber

41. JavaOneJavaOne
41
Stage/thread communication
Pipes
Synchronized structures + wait/notify
Blocking queues

42. JavaOneJavaOne
42
Stage/thread communication
 PipedOutputStream, PipedInputStream
 Advantages:
 Can layer the wide variety of stream classes to do transforms, etc.
 Synchronization, waiting, notification is handled for you
 A full output pipe automatically forces a producer that is too fast to slow down
 Buffer size expressed in bytes – great if you want to strictly control the memory
used
 Disadvantages
 All objects must be serialized – not so great if you are not ultimately writing to
disk
 Must be 1:1 – one producer and one consumer (per pipe)
 Push only

43. JavaOneJavaOne
43
Producer w/ Pipes
public class Stage1 implements StageWorker{
private PipedOutputStream outPipe;
Stage1(Stage successor) {
outPipe = new PipedOutputStream(successor.inPipe);
}
…
private void sendData(WorkUnit workUnit) {
ObjectOutputStream outObjStream = new ObjectOutputStream(outPipe);
workUnit.writeObject(outObjStream);
}
}

44. JavaOneJavaOne
44
Consumer w/ Pipes
public class Stage2 implements StageWorker {
public PipedInputStream inPipe;
Stage2() {
inPipe = new PipedInputStream();
}
…
private WorkUnit receiveData() {
ObjectInputStream inObjStream = new ObjectInputStream(inPipe);
WorkUnit workUnit = new WorkUnit();
workUnit.readObject(inObjStream);
return workUnit;
}
}

45. JavaOneJavaOne
45
Stage/thread communication
 Synchronized structures + wait/notify
 Advantages:
 Allows multiple consumers to access upstream data at their own pace,
i.e. they can pull their data instead of having it pushed to them
 Consumers determine what data they want
 Disadvantages:
 Producers and consumers must synchronize on the same object
 Must manage synchronization, waiting, notification yourself
 This includes making the producer wait if consumers are falling behind
 Cannot read while writing and vice-versa

46. JavaOneJavaOne
46
Producer w/ Synchronized Access
public class Stage1 implements StageWorker {
public LinkedList<WorkUnit> outList ;
Stage1(Stage successor) {
outList = new LinkedList<WorkUnit>(1000);
}
…
private void sendData(WorkUnit workUnit) {
synchronized(outList) {
outList.add(workUnit);
outList.notifyAll();
}
}

47. JavaOneJavaOne
47
Consumer w/ Synchronized Access
public class Stage2 implements StageWorker {
private ArrayList<WorkUnit> inList;
public int currentPos = 0; // This needs to be adjusted for removals
Stage2(Stage predecessor) {
inputList = predecessor.outList;
}
…

48. JavaOneJavaOne
48
Consumer w/ Synchronized Access
private WorkUnit receiveData() {
WorkUnit workUnit = null;
synchronized(inList) {
while (currentPos >= inList.size()) {
try {
inList.wait(1000);
} catch (InterruptedException ie) {}
}
workUnit = inList.get(currentPos++);
}
return workUnit;
}
}

49. JavaOneJavaOne
49
Stage/thread communication
 ArrayBlockingQueue, LinkedBlockingQueue
 Use the capacity setting to control how far ahead a producer can get before it
blocks
 Advantages:
 Synchronization, waiting, notification is handled for you
 A full output queue automatically forces a producer that is too fast to slow down
 Can read while writing and vice-versa
 Capacity is expressed as a # of objects – great if you want to limit how many
objects the producer gets ahead of the consumer
 Disadvantages:
 Memory use can vary if object sizes are highly variable
 There is still a synchronization cost to put/take
 Push only

50. JavaOneJavaOne
50
Producer w/ ArrayBlockingQueue
public class Stage1 implements StageWorker {
private ArrayBlockingQueue outQueue ;
Stage1(Stage successor) {
outQueue = successor.inQueue;
}
…
private void sendData(WorkUnit workUnit) {
outQueue.put(workUnit);
}
}

51. JavaOneJavaOne
51
Consumer w/ ArrayBlockingQueue
public class Stage2 implements StageWorker {
public ArrayBlockingQueue<WorkUnit> inQueue;
Stage2() {
inQueue = new ArrayBlockingQueue<WorkUnit>;
}
…
private WorkUnit receiveData() {
WorkUnit workUnit = inQueue.take();
return workUnit;
}
}

52. JavaOneJavaOne
52
Stage/thread communication
Recommendations
 Use ArrayBlockingQueue except where there are multiple
consumers
LinkedBlockingQueue is not recommended, as it is unbounded
by default, and has higher memory utilization
If multiple consumers, have the consumers pull their data
using synchronization + wait/notify

53. JavaOneJavaOne
53
Stage/thread communication
 For good performance, it is critical to minimize the
synchronization costs of inter-stage communication
The key is maximizing the unit of work passed between
stages, i.e. buffering
 Batch up smaller units of work into buffers of larger ones
The buffers are passed between stages instead of the original
units of work
Buffering can decrease responsiveness
The key here is to flush the buffers whenever any thread must
wait

54. JavaOneJavaOne
54
Stage/thread communication
When using ArrayBlockingQueue:
Wrap in a custom queue that buffers objects and puts the
buffers to the underlying queue, instead of the actual objects
Take from the queue then gets the buffers, and processes
them
When using synchronized access:
Producer buffers objects, then adds full buffers to the
synchronized common data structure
Consumers read full buffers from the synchronized common
data structure

55. JavaOneJavaOne
55
Waiting/Flushing
Waiting indefinitely is dangerous!
Try to NEVER wait on anything indefinitely
If you are waiting for another thread to do something, and it
dies, you will be waiting forever
So, use loops+timeouts whenever possible
 When wake up, check if anyone wants you to stop, process a more
urgent request, etc.
 If not, re-issue request

56. JavaOneJavaOne
56
Waiting/Flushing
Interrupting threads
Dangerous to rely on interrupts to break indefinite waits
Who will send the interrupt and how will they know it’s needed?
 Interrupting thread must have a handle to the interrupted thread

57. JavaOneJavaOne
57
Waiting/Flushing
Interrupting threads
Could interrupt an I/O in process. The results of this are
unpredictable:
 InterruptedIOException
 I/O could have partially completed
 ClosedByInterruptException (if I/O was through a channel)
 Ignored
 For example, if thread is blocked waiting on a socket
(ServerSocket.accept)
Even if your module handles interrupts correctly – can you be
sure all called modules do?
Best to avoid using interrupts!

58. JavaOneJavaOne
58
Waiting/Flushing
Flush all buffers before waiting
Otherwise already processed data can get “stuck” in the
pipeline
Optimization: First wait with a short timeout, and then if still no
data, flush at that point

59. JavaOneJavaOne
59
Stopping
Requesting stop via thread interruption
If thread is interrupted, it assumes it should stop
 This is how ExecutorService.shutdownNow() and Future.cancel()
signal threads to stop
 So threads that stop this way will be stoppable if run as part of an
ExecutorService
 BUT:

60. JavaOneJavaOne
60
Stopping
Requesting stop via thread interruption
Class has to run as a thread, and it needs to know it is running
as a thread
 If thread is sleeping or waiting, InterrruptException will be thrown
which will clear the interrupted state
 The catch of InterruptedException has to signal the class to stop or has to
re-interrupt the thread
 Has all the problems mentioned earlier if an I/O is interrupted
 No granularity in the stop request
 Cannot distinguish between stop immediately and stop gracefully
So this is not recommended!

61. JavaOneJavaOne
61
Stopping
Requesting stop via flag objects
Have a stop request object that indicates thread should stop
Classes check regularly, and always before sleeping, waiting,
I/O, putting/getting from a blocking queue, etc.
Works the same regardless of whether the class is running as
a thread or not
Stop request object can indicate different types of stop: stop
immediate, stop gracefully (finish work in progress), etc.
 Stopping gracefully is important in a pipeline – in most cases, you
don’t want to lose the work already completed!

62. JavaOneJavaOne
62
Stopping
Have a stop request object that indicates thread should stop
Can have a single stop request object for the entire pipeline
that all threads monitor
 But this only allows the entire pipeline to be stopped
For finer granularity of stop control, have a pipeline stop
request object with references to stage/thread stop request
objects
Setting of stop request object must be synchronized – but this
does not occur very often

63. JavaOneJavaOne
63
Stopping
Have a request stop object that indicates thread should stop
Reading of the stop request object will occur frequently
Reading stop request object does not need to be synchronized
for most pipeline applications
 It is not critical to see a stop request instantaneously, and
synchronization doesn’t guarantee this anyway
 The stop request is typically a boolean or enum, so updates of it are
atomic
 There will be enough other synchronization boundaries crossed to
guarantee the stop request object modification will be seen
 Puts/gets to blocking queues, synchronized access to other shared
structures, etc.
 Synchronizing on write but not on read is known as Asymmetrical Locking

64. JavaOneJavaOne
64
Stopping
Detecting a stage/thread has actually stopped
Could use Thread.isAlive() to detect if a thread has
stopped
This requires a handle to the thread
Presumes the class is running as a thread
Won’t work for higher abstractions, such as a stage or
pipeline

65. JavaOneJavaOne
65
Stopping
Detecting a stage/thread has actually stopped
Better to make this abstract
Separate Runnable class from class that does the real work
Runnable class invokes the work class
The Runnable class has an “isRunning” object that tracks the
run status
All threads use this common Runnable class
Guarantees all threads have a common mechanism for
tracking run status
Can also be used to guarantee common handling for uncaught
exceptions

66. JavaOneJavaOne
66
Stopping
Detecting a stage/thread has actually stopped
The status of a stage is the logical-AND of the status
of the individual threads of that stage
The status of a pipeline is the logical-AND of the
status of the stages
If a graceful stop is requested, stages must wait for
their predecessor stages to actually stop before they
can stop
This allows in-flight data to be processed

67. JavaOneJavaOne
67
Stopping
Unexpected thread termination
Just set the stop request object to bring down all related
threads
Easiest and safest thing to do is bring down entire pipeline

68. JavaOneJavaOne
68
Stopping
Returning control to the user
Controller or highest level pipeline thread must make sure all
threads are truly stopped before returning
Strange things can happen if you try to start a pipeline and
some threads from its previous incarnation are still running!
Could call thread.join()
 But must have handle to all threads, and can’t be responsive to
other requests
Better for the controller to query the isRunning flag of all thread
classes and wait for all to be stopped.

69. JavaOneJavaOne
69
Example: buffering, flushing, stopping
work()
{
while (!checkForStopRequests()) {
int outBufSize = 100;
ArrayList<WorkUnit> outBuf = new ArrayList<WorkUnit>(bufSize);
ArrayList<WorkUnit> inBuf = inQueue.poll(1000,TimeUnit.MILLISECONDS);
// If have to wait for input, flush output
if (inBuf == null) {
if (outBuf.size() > 0) {
flushOutput();
}
}

70. JavaOneJavaOne
70
Example: buffering, flushing, stopping
else {
for (WorkUnit workUnit : inBuf) {
// process workUnit
…
outBuf.add(workUnit);
if (outBuf.size() > outBufSize) {
flushOutput();
}
}
}
}

71. JavaOneJavaOne
71
Example: buffering, flushing, stopping
flushOutput() {
boolean success = false;
while (!success && !checkForStopRequests()) {
success = outQueue.offer(outBuf,1000,TimeUnit.MILLISECONDS);
}
}
Boolean checkForStopRequests() {
// check for stop requests. Basic handling is:
if (stop requested) {
stageRunnableObj.isRunning = false;
return true; // stop this stage
}
return false;
}

72. JavaOneJavaOne
72
Uncaught exception handling
 Proper handling for uncaught exceptions is absolutely critical for a
pipeline!
 Uncaught exceptions are typically the leading cause of unexpected
thread termination
 Since all threads are dependent on others, if one goes down, the
entire pipeline can hang!
 A hung pipeline can be hard to detect
 Is it hung, or just in-between bursts of traffic?
 Proper monitoring statistics can help detect a hung pipeline
 It can take a while before you can be sure
 Auto-aborting a pipeline that is perceived to be hung will inevitably abort too soon in
certain situations
 So typically requires human intervention
 The best way to handle a hung pipeline is to prevent it!

73. JavaOneJavaOne
73
Uncaught exception handling
By default, uncaught exceptions are handled by the default
uncaught exception handler
This just writes the stack trace to standard error
This is rarely acceptable!
 Stack trace may be lost
 If class tracks its run status, this will be inaccurate after an uncaught
exception: run status will still be ‘running’
 What about threads that are dependent on this thread?
Don’t let the default happen to you!

74. JavaOneJavaOne
74
Uncaught exception handling
 Ensure you have handling for all uncaught exceptions
 Keep it simple: do what must be done, but no more
 Complex handling for uncaught exceptions could trigger other uncaught
exceptions, which can be extremely difficult to debug!
 And since uncaught exceptions don’t occur often, the handler probably has not
been tested as much as other modules
 All handlers, at a minimum, should:
 Log the error and stack trace somewhere durable and secure
 Alert the user so the error will be noticed
 If the thread class has a run status, the run status should be set to stopped,
error, or some equivalent
 Requires access to the thread class state

75. JavaOneJavaOne
75
Uncaught exception handling
 Ideally, the handler should also alert dependent threads
 Can notify controller or parent, which can:
 Restart the thread that got the uncaught exception
 Not recommended! The pipeline could be in an indeterminate state
 Stop the pipeline (safest)
 Set the stopping request object to stop the pipeline
 Since all stages/threads are monitoring, this will bring down the pipeline
 Pipeline won’t hang!
 Controller doesn’t need an explicit message to know thread has
died, if it is monitoring the run status of the pipeline threads
 In fact, the controller doesn’t even need to get involved –
 Just have the uncaught exception handler set the stop request object to stop

76. JavaOneJavaOne
76
Uncaught exception handling
Where to handle?
Three primary possibilities:
Custom uncaught exception handler
Catch Throwable() in run() method
Finally block in run() method

77. JavaOneJavaOne
77
Uncaught exception handling
Custom uncaught exception handler
 If a uncaught exception handler is declared and is in scope,
the default uncaught exception handler is NOT invoked
Determine scope:
Per JVM
Per Thread group
Per Runnable class

78. JavaOneJavaOne
78
Uncaught exception handling
Custom uncaught exception handler – per JVM scope
Logging the error and alerting the user is about all that can
be done
Since this is per-JVM, it does not have access to any specific
thread classes or thread groups
Set via the static Thread method
setDefaultUncaughtExceptionHandler
Thread.setDefaultUncaughtExceptionHandler(
Thread.UncaughtExceptionHandler ueh)
 ueh is a class that must implement uncaughtException(Thread thr,
Throwable the)

79. JavaOneJavaOne
79
Uncaught exception handling
Custom uncaught exception handler – per Thread group
Can guarantee standard handling for a group of threads by
having their Thread instance use the same custom
ThreadGroup class
Custom ThreadGroup class extends ThreadGroup and
overrides the uncaughtException method
This is a good choice for a “default” exception handler
Allows for common code for the thread group, but doesn’t have
access to individual Runnable classes

80. JavaOneJavaOne
80
Uncaught exception handling
Custom uncaught exception handler – per Thread class
Can have access to the Runnable class, so the handler can log
details about the class structure, set run status, etc. before
exiting
Guaranteed to be invoked
Set via the Thread method setUncaughtExceptionHandler
setUncaughtExceptionHandler(
Thread.UncaughtExceptionHandler ueh)
 ueh is a class that must implement uncaughtException(Thread thr,
Throwable the)
 Simplest design is to have ueh be the Runnable class

81. JavaOneJavaOne
81
Uncaught exception handling
Catch Throwable() in run() method
In general, this is NOT recommended
 Bad form
 If you do this, do it outside any loops in the thread run method, i.e.
at the highest level possible.
 Otherwise, the exception will not trigger the thread to exit, which can
cause very strange thread conditions!
 Thread can be “alive”, but constantly cycling through errors, or hanging

82. JavaOneJavaOne
82
Uncaught exception handling
Finally block in run() method
Have handled flag that is set to true at end of try block and
every catch block
!handled = uncaught exception
Advantage: Can have handling customized to the invoked
class
 Can set run status and stop request
Disadvantages
 Every catch block must set “handled” flag or the checked exception
will be treated as unhandled
 Default uncaught exception handler still invoked

83. JavaOneJavaOne
83
Uncaught exception handling
 Critical threads
 Some threads may be critical to your application
 If they stop unexpectedly, the application will not function
 Best to stop the entire application when this occurs
 The uncaught exception handling for the thread should call
System.exit(<non-zero value>)
 If the ThreadGroup uncaught exception handler is guaranteed to be invoked,
then can have a separate critical thread ThreadGroup
 All critical threads are part of this group
 The group handler calls system.exit()

84. JavaOneJavaOne
84
Example using finally block
public class StageRunnable implements Runnable {
boolean isRunning;
StageWorker worker;
boolean handled;
StageRunnable(StageWorker worker) {
this.worker = worker;
handled = false;
}

85. JavaOneJavaOne
85
Example using finally block
run() {
try {
isRunning = true;
worker.work();
handled = true;
}
finally {
if (!handled) {
// uncaught exception!
isRunning = false;
worker.setStopRequest(true);
// log error and stack trace to error file
}
}
}
}

86. JavaOneJavaOne
86
Example using custom handler
public class StageRunnable implements Runnable,
Thread.uncaughtExceptionHandler {
boolean isRunning;
StageWorker worker;
StageRunnable(StageWorker worker) {
this.worker = worker;
}

87. JavaOneJavaOne
87
Example using custom handler
public void uncaughtException {Thread thr, Throwable the)
isRunning = false;
worker.setStopRequest(true);
// log error and stack trace to error file
}
}
Add to where the actual thread is created:
stageThread.setUncaughtExceptionHandler(stageRunnable);

88. JavaOneJavaOne
88
Shutdown hooks
 Can declare a special thread to run at shutdown of the JVM
 This is added via System.addShutdownHook(Thread hook)
 Typical usage would be to delete/close files or other resources that
didn’t get deleted/closed normally
 Shutdown hooks should be quick!
 Users don’t like waiting for the JVM to stop
 If the computer is stopping, there may not be much time before the
process is killed

89. JavaOneJavaOne
89
Shutdown hooks
 Ideally, only use to clean up resources for abnormal JVM termination
 Then in most cases, there is nothing for the shutdown hook to do
 Multiple shutdown hooks can be declared
 Order of execution is not guaranteed
 Make sure there are no dependencies between shutdown hooks
 Eliminate the dependencies, or
 Combine dependent shutdown hooks into one hook
 Shutdown hook threads run concurrently with any other threads still
running
 For example, they will run concurrently with any daemon threads still running

90. JavaOneJavaOne
90
Daemon Threads
Daemon threads exist only as long as there is one user
(non-daemon) thread running
They will stop abruptly when the last user thread stops
They will not stop on their own any sooner
It’s best to explicitly ask them to stop when they are no
longer needed

91. JavaOneJavaOne
91
Daemon Threads
 Daemon threads are useful for threads that do ‘side-work’ and you
don’t want to keep the JVM running if all user threads are stopped
 For example, a performance monitor thread
 Very useful for a helper module that is embedded in another and
creates extra threads
 Enclosing module is unaware of these extra threads
 If the last enclosing module thread stops and helper module isn’t
notified, the extra helper module threads will keep the JVM from
stopping unless they are daemon threads

92. JavaOneJavaOne
92
Daemon Threads
Daemon threads should always do their own cleanup
If cleanup is necessary, put in finally block
Could add a shutdown hook to do cleanup for the daemon
threads
But there is no guarantee that it will run after the daemon
thread stops

93. JavaOneJavaOne
93
Summary
A well-balanced pipeline can increase throughput,
responsiveness, and processor utilization significantly
Proper thread control and communication is the key to a
successful pipeline
Buffering between stages can reduce synchronization costs
significantly
A framework for stopping threads and consistent handling
of uncaught exceptions is essential to avoid hanging

94. JavaOneJavaOne
94
IBM at Oracle Open World / JavaOne
 See IBM in each of these areas throughout the OOW event:
• JD Edwards Pavilion at the InterContinental Hotel Level 3… IBM Booth #HIJ-012
• Moscone West ….Guardium Booth #3618
• Java One Expo Floor at the Hilton San Francisco Union Square…IBM Booth #5104
• ILOG at the Java One Expo at the Hilton San Francisco Union Square….IBM Booth #5104
 Meet with IBM experts at our “Solution Spotlight Sessions” in Moscone South, Booth #1111.
 Access a wealth of insight including downloadable whitepapers & client successes at
ibm.com/oracle.
For the most current OOW updates follow us on Twitter at www.twitter.com/IBMandOracle

95. Phillip Koza
pkoza@us.ibm.com
JavaOneJavaOne Thank YouThank You