HWRS2015_Diermanse_Paper_51-Final

An Efficient Sampling Method for Fast and Accurate Monte Carlo
Simulations
F.L.M. Diermanse
Expert Researcher, Deltares, Delft, The Netherlands
E-mail: ferdinand.diermanse@deltares.nl
D.G. Carroll
Director, Don Carroll Project Management, Brisbane, Australia
E-mail: don.carroll@optusnet.com.au
J.V.L. Beckers
Senior Advisor/Researcher, Deltares, Delft, The Netherlands
E-mail: joost.beckers@deltares.nl
R. Ayre
Project Leader, Aurecon, Brisbane, Australia
E-mail: Rob.Ayre@aurecongroup.com
Currently, there are two published Monte Carlo Simulations (MCS) methods for estimating design
floods in Australia: The Cooperative Research Centre – Catchment Hydrology (CRC-CH) method and
the Total Probability Theorem (TPT) method. The CRC-CH method uses variable storm durations,
which makes it an attractive option compared with the TPT method. However, the CRC-CH method
suffers from a large variance in design flood estimates for large to extreme events, ie events with low
annual exceedance probabilities (AEP). This means two successive MCS runs may provide
significantly different design flows, which makes the method unreliable for large to extreme events.
Decreasing this variance to an acceptable level may require millions of hydrological model simulations.
We introduced an alternative sampling technique, known as ‘importance sampling’, in the CRC-CH
method to address this shortcoming. In a pilot study, carried out as part of the Brisbane River
Catchment Flood Study, we demonstrate that this results in a significantly reduced variance in design
flood estimates for large to extreme events. At the same time, computation times are significantly
shortened. This creates opportunities for greater use of the CRC-CH method and similar Monte Carlo
methods in applications where they were previously considered infeasible, such as dam design.
1. INTRODUCTION
The State of Queensland initiated a comprehensive hydrologic assessment as part of the Brisbane
River catchment flood study in response to the devastating floods in January 2011 and subsequent
recommendations of the Queensland Floods Commission of Inquiry (QFC, 2012). The goal of the
study is to produce a set of competing methods for estimating design floods for the Brisbane River
catchment. One of the proposed methods is based on Monte Carlo Simulations (MCS), sometimes
referred to as the ‘joint probability approach’. The method has the advantage over more traditional
approaches in that it provides a more realistic representation of all the physical processes that
contribute to flood events. The Monte Carlo Simulations framework developed for the Brisbane River
catchment Flood Study describes probabilities, mutual correlations and physical interactions of all
relevant factors for flood risk in the catchment. These factors include rainfall depth, spatial and
temporal distribution of rainfall, antecedent moisture conditions, initial reservoir volumes and ocean
water levels. The various components of the framework are described in more detail in Diermanse et
al (2014).
The choice of sampling method is crucial for the Monte Carlo Simulations. Several methods were
considered in the Brisbane River catchment flood study, such as the Cooperative Research Centre –
Catchment Hydrology (CRC-CH) method (Rahman et al, 2002) and the Total Probability Theorem
(TPT) method (ARR, 2013). The CRC-CH method is referred to as conceptually superior to TPT

An Efficient Sampling Method for Fast and Accurate Monte Carlo Simulations Diermanse
HWRS 2015 Diermanse, Carroll, Beckers and Ayre 2 of 8
method by Mirfenderesk et al (2013) and considered to be more suitable for volume-sensitive
applications (Carroll, 2012). A problem with the CRC-CH method, however, is that it suffers from a
large variance in design flood estimates for large to extreme events, i.e. events with low annual
exceedance probabilities (AEP). This means two successive runs may provide significantly different
design flows, which makes the method unreliable. For this reason, Rahman et al (2002) considered
their method to be appropriate only for annual exceedance probabilities of 1 in 100 and higher.
Decreasing the variance to an acceptable level for low AEP-values may require millions of
hydrological model simulations. The CRC-CH method is therefore considered unsuitable for extreme
floods by Mirfenderesk et al (2013).
The problem of large variances in probability estimates for extreme events is a well-known drawback
of ‘crude’ Monte Carlo Sampling (Dawson and Hall, 2006), which is the applied sampling method in
the CRC-CH approach. Crude sampling means the stochastic variables are sampled directly from their
respective distribution functions. This means that if annual maximum events are simulated, the 1 in
100,000 AEP peak discharge is exceeded on average once every 100,000 simulations. Hence, a
multitude of 100,000 simulations needs to be carried out to provide a reliable estimate of the 1 in
100,000 AEP peak discharge. Completing such a large number of simulations is usually untenable.
One of the methods to reduce the variance in Monte Carlo estimates is ‘importance sampling’
(Diermanse et al, 2014, Sadowski and Bucklew, 1990, Siegmund, 1976). In a pilot study we
demonstrate the application of importance sampling in the CRC-CH approach. First, Monte Carlo
sampling techniques, including importance sampling, are explained.
2. MONTE CARLO SAMPLING
This section describes the computation of the exceedance probability of a discharge threshold level, q,
with Monte Carlo Simulation. Define vector X as the set of all stochastic variables involved in the
Monte Carlo Simulation and define x as a sample, or realisation, of X. Each x characterises a
synthetic flood event, which means for each x a peak discharge can be computed by means of a
hydrologic simulation model. The resulting peak discharge Q can thus be written as a function of x:
Q=Q(x). To compute the exceedance probability of a discharge threshold level q, we need to quantify
the combined probability of all realisations, x, for which Q(x)>q. In formula:
   
 
   
:
Ievent Q q
Q q
P Q q f d f d  

   X X x
x x x
x x x x (1)
Where Pevent is the exceedance probability of threshold level q in an individual event, fX is the joint
probability density function (pdf) of x and I is the indicator function: I=1 if Q>q, I=0 otherwise. Eq. (1)
provides exceedance probabilities of discharge levels per event. In order to derive annual exceedance
probabilities, the number of simulated events per year, , needs to be taken into account as follows:
     1 1year event eventP Q q P Q q P Q q

         (2)
The integral of eq. (1) is generally too complex to compute analytically, which is why joined probability
methods like Monte Carlo Simulations are carried out to approximate it.
2.1. Crude Monte Carlo Sampling
The essence of the crude Monte Carlo Simulation method is to repeatedly sample x from density
function fX and to subsequently verify for each sample (through model simulations) if level q is
exceeded. The fraction of samples for which q is exceeded is an estimate of the exceedance
probability of q and, hence, an estimate for the integral of eq. (1). This estimate is equal to:
   
1
1ˆ I i
N
event q q
i
P Q q
N 

   (3)

Where N is the number of samples of x (ie the number of simulated events) and qi is the derived peak
discharge in the ith simulated event. The probability estimate according to eq (3) is proportional to the
number of simulated events in which threshold level q is exceeded. For large to extreme discharges,
the number of exceedances will be relatively small. For example, if 5000 simulations are carried out,
the peak discharge with exceedance probability 1 in 1000 is expected to be exceeded 5 times. Due to
the random nature of Monte Carlo Simulations, the observed number of exceedances may also be 3
or 6. This means relative differences in probability estimates of subsequent Monte Carlo experiments
may easily be a factor two. The random character of Monte Carlo simulations thus leads to relatively
large variations in the resulting probability estimates for high threshold levels of q. An increase in the
simulated number of exceedances of q, in order to decrease the undesired variation in probability
estimates, could be obtained by increasing the number of samples (N). However, this results in a
proportional increase in computation time, which is often untenable. Importance sampling may be an
attractive alternative in that case.
2.2. Monte Carlo importance sampling
Importance sampling means an alternative sampling function, hx, is used instead of the actual density
function, fx, to generate event samples. The function hx should be chosen in such a way that the
proportion of sampled “extreme events” in the Monte Carlo Simulations increases to an extent that
reliable probability estimates can be provided for extreme discharges. The increase in the number of
sampled extreme events essentially means more information is provided on these events, which
reduces the variability in the probability estimates. Application of importance sampling implies a bias in
the probability estimate is introduced if the probability estimator of eq. (3) were to be used. An
alternative estimator is therefore required in order to provide unbiased estimates. For this purpose,
equation (1) is rewritten as follows:
       
 
   
I Ievent Q q Q q
f
P Q q f d h d
h       
  
    
  
 
X
X Xx x
Xx x
x
x x x x
x
(4)
Since samples are now taken from function hx, an estimator for this probability is provided by:
   
 
 1
1ˆ I i
N
X i
event q q
i X i
f
P Q q
N h

  
x
x
(5)
In which xi is the ith sample of x and qi is the derived peak discharge in the ith simulated event that was
generated with sampling function hx. Eqs. (3) and (5) are both estimators for the exceedance
probability of threshold q, but the estimator of eq. (5) uses a different set of event simulations, i.e. a
set generated with importance sampling function hx. This is why there is a difference in formulation of
the two estimators via the factor ci =f(xi)/h(xi). This factor needs to be computed for each sampled
event xi. The reason is that the probability of sampling xi was increased by a factor h(xi)/f(xi)/h(xi) in
the importance sampling procedure, i.e. increased with a factor 1/ci. This needs to be compensated for
in the probability estimate through multiplication of a factor ci, to prevent the introduction of a bias in
the probability estimate. So, for example, if the probability of sampling a specific event has been
increased by a factor 10 as a result of importance sampling, the contribution of this event to the
estimated exceedance probability of discharge level q needs to be divided by a factor 10.
3. PILOT STUDY
A pilot study was carried out for a hypothetical catchment to assess the potential benefits of
importance sampling. The hypothetical catchment was modelled after Rahman et al (2002). The
relatively straightforward hydrological model of the (hypothetical) catchment allows for millions of
hydrological model runs within a matter of minutes and as such provides excellent opportunities for
analysis of the accuracy and variability of the MCS design flow estimates.

3.1. Catchment and rainfall runoff model
The area of the catchment is set equal to 100 km2. Runoff percentages during a storm event are
influenced by six factors: rainfall duration, rainfall depth, rainfall temporal pattern, spatial distribution of
rainfall, initial losses and continuing losses. Four of these six factors are modeled as stochastic
variables: rainfall duration, rainfall depth, rainfall temporal pattern and initial losses. For the spatial
pattern, a uniform distribution is assumed and the continuing loss is assumed to be 2.5 mm/hr,
independent of the storm event. The net rainfall, i.e. the proportion of the rainfall available for runoff, is
derived by first subtracting the initial loss from the rainfall depth and subsequently subtracting the
continuing loss from the “remaining” rainfall. The discharge is modelled as follows:
m
S kQ (6)
In which S is a storage component of the net rainfall (mm), Q is the discharge at the catchment outlet
(mm/hr) and k and m are model parameters. Higher values of k and m generally result in a decrease in
the peak discharge and earlier timing of the peak. In the current studies, k=0.2 and m=1.
3.2. Input statistics
The duration of storm events is assumed to be exponentially distributed with a mean of 20 hours. The
statistics of the rainfall depth are described with Intensity-Frequency-Duration (IFD) relations (Table 1).
Values for 1 - 1/100 AEP and 5 mins – 72 hours were obtained from the BoM website for location
Brisbane (ARR, 1987). Values for AEP < 1 in 100 and values for durations>72 hours were obtained
through linear extrapolation. Note: The IFD curves of Table 1 have been derived by the BoM for
rainfall bursts. In the current application these will be used to sample rainfall depths for events. In real-
world application this should not be done, as event statistics are different from burst statistics. For the
purpose of this paper this assumption does not pose a problem as the derived statistics only serve to
show the potential of the importance sampling procedure.
Table 1 Assumed IFD-curves (rainfall depth in mm)
Annual exceedance probability (1 in N)
Duration 1 2 5 10 20 50 100 1000 104
105
106
1 hr 36.0 46.7 61.2 70.0 81.7 97.5 110.0 150.0 190.0 230.0 270.0
2hrs 45.2 58.8 77.6 89.2 104.4 125.2 141.4 194.0 246.0 298.0 350.2
3 hrs 50.4 66.0 87.3 100.8 118.2 141.9 160.5 219.0 279.0 339.0 399.3
6 hrs 60.6 79.8 106.2 122.4 144.0 174.0 197.4 270.0 348.0 420.0 497.4
12 hrs 75.6 98.4 132.0 152.4 180.0 217.2 246.0 336.0 432.0 528.0 620.4
24 hrs 98.4 127.2 170.4 196.8 232.8 278.4 316.8 432.0 552.0 672.0 796.8
48 hrs 129.6 168.0 220.8 254.4 297.6 360.0 408.0 576.0 720.0 864.0 1012.8
72 hrs 144.0 187.2 244.8 280.8 331.2 396.0 453.6 648.0 792.0 936.0 1116.0
96 hrs 153.6 201.6 268.8 307.2 355.2 432.0 480.0 662.4 835.2 1017.6 1200.0
120 hrs 168.0 216.0 288.0 324.0 384.0 456.0 516.0 696.0 888.0 1068.0 1260.0
168 hrs 184.8 235.2 302.4 352.8 420.0 487.2 554.4 756.0 957.6 1159.2 1360.8
Rainfall temporal patterns are simulated using a three layer multiplicative cascade model i.e. each
event is subdivided into 23 = 8 time intervals (see Figure 1) of equal size in which the rainfall depth
remains constant. This model is widely used in Australia and shown with appropriate parameterization
that it can statistically represent historical temporal patterns (Hoang, 2001, Carroll and Rahman,
2004). The stochastic simulation of the rainfall distribution over the time intervals is simulated with 23-1
= 7 stochastic variables W1…W7, which are assumed to be uniformly distributed between 0.2 and 0.8.

Variable W1 determines the proportion of the total rainfall in the first half of the event (see Figure 1).
This means the second half of the rainfall event receives a proportion of 1-W1 of the total rainfall. In
similar fashion, variables W2…W7 determine the proportion of rainfall in the first half of other time
intervals, as shown in Figure 1. The third layer of Figure 1 shows the 8 time intervals of the cascade
model. The proportion of the total rainfall that falls in the first interval is equal to W1*W2*W4. The
proportion of the total rainfall that falls in the second interval is equal to W1*W2*(1-W4) etc.
first layer
second layer
third layer W4 1-W4 W5 1-W5 W6 1-W6 W7 1-W7
W1 1-W1
W2 1-W2 W3 1-W3
time
Figure 1 Three-layer multiplicative cascade model
Initial losses are assumed to have a four-parameter beta-distribution, after Ilahee et al (2001). The first
two parameters of this distribution are the minimum, Dmin, and maximum, Dmax, values and these are
taken equal to 0 mm and 100 mm respectively. Shape parameters  and  are taken equal to 2 and 5
respectively.
3.3. Sampling procedure
The sampling procedure, modelled after Rahman et al, (2002), can be summarized as follows:
1. Sample the storm event duration from the exponential distribution function;
2. Sample the rainfall depth, conditioned on duration;
3. Sample the rainfall temporal pattern from the multiplicative cascade model;
4. Sample the initial loss from the beta-distribution;
5. Run the rainfall runoff-model to derive the peak discharge at the catchment outlet.
The procedure is repeated N times, in order to simulate N storm events. The average number of storm
events per year, , is taken equal to 5, which means in total N/ years are simulated. Importance
sampling is applied in step 2. For comparison, additional simulations with crude Monte Carlo sampling
are carried out. In the latter case, a probability, Pe, is sampled in step 2 from the standard uniform
distribution. Pe is the probability of exceedance of the rainfall depth for an individual event, given the
rainfall duration. Since multiple () events per year are considered, Pe is not the same as the annual
exceedance probability (AEP). The average return interval (ARI) of this event is equal to:
 
1
log 1
rain
e
ARI
P
 

(7)
Note that we use ARI in the formulation instead of the more commonly used AEP because this makes
the subsequent equations easier to interpret. The sampled value of the ARI is used to derive the
corresponding rainfall depth from Table 1. Eq. (7) implies the distribution function of the ARI of the
rainfall depth in the crude Monte Carlo procedure is equal to:
   
1
ARI exprainF x P x
x
 
    
 
(8)
The corresponding density function is:
  2
1
exp
x
f x
x


 
 
  (9)

The objective of the importance sampling procedure is to have a more or less equal representation
over the entire range of relevant peak discharges. From initial model simulations, we observed that the
scattered relation between rainfall depth and peak discharge shows a more or less linear trend (the
scatter being caused by other factors like rainfall duration, temporal patterns and initial losses).
Furthermore, there is a near linear relationship between the rainfall depth and the logarithm of the ARI
of the rainfall depth, log(ARIrain). This means the (scattered) relation between log(ARIrain) and the
resulting peak discharge is also approximately linear. So, in order to have a procedure that results in a
(nearly) uniformly distributed set of peak discharges samples, a sampling strategy is proposed in
which log(ARIrain) is uniformly distributed:
   log ARI U ,rain a b (10)
This sampling strategy implies that the expected number of sampled events with an ARI of the rainfall
depth between 10k and 10k+1 years is the same for each value of k. For example, the expected number
of samples with ARI between 10 and 100 years (i.e. AEP between 1 in 10 and 1 in 100) is the same as
the expected number of samples with ARI between 100 and 1,000 years (i.e. AEP between 1 in 100
and 1 in 1,000). In crude Monte Carlo sampling there would be approximately ten times as many
samples in the first interval compared to the second interval. This shows the importance sampling
results in a more balanced representation of moderate and extreme events. Parameters a and b of eq.
(10) are taken equal to the limits of the ARIs for which rainfall intensities are given in the IFD relations
of Table 1. This means a = log(1) and b = log (106). This will provide samples of rainfall depths over
the entire range for which IFD curves are available. Equation (10) translates into the following
importance sampling distribution function for ARIrain:
   
 
 
log
; lograin
x a
H x P ARI x a x b
b a

    

(11)
The corresponding density function is:
 
 
1
;h x a x b
x b a
  

(12)
The importance sampling correction factor, f/h, of eq. (5) is thus equal to:
 
 
 
 
1
exp
; ln
b a
f x x
a x b
h x x


 
  
    (13)
This correction factor needs to be quantified for each simulated event in the Monte Carlo procedure,
by replacing the value of x in equation (13) with the sampled value of the ARI of the rainfall depth.
4. RESULTS
The crude Monte Carlo and importance sampling approaches were applied using 20,000 and 5,000
simulated storm events respectively. The procedure was repeated 100 times to assess the variability
in the Monte Carlo estimates, and the results are shown in Figure 2. Each blue line shows the results
of a single crude Monte Carlo run, each red line the results of a run with importance sampling.
Additionally, a crude Monte Carlo run with 100 million simulated storm events was carried out (green
dots), which can be expected to provide very accurate results and therefore serves as a benchmark.
Note a run with 100 million samples is generally untenable in practice, but the simplicity of the
hydrologic model of the current case study enables such a large number of simulation runs.
Two disadvantages of the crude Monte Carlo sampling method are clearly demonstrated by the blue
lines in this graph:

1. For annual exceedance probabilities that are smaller than the reciprocal of the number of
simulated years (ie smaller than 1 in 4000 in this case: 20,000 simulations with 5 events per
year) the method provides no results.
2. The variation in estimated design discharges increases with decreasing value of the annual
exceedance probability. Design flow estimates for an annual exceedance probability of 1 in
4000 range from 750 m3/s to 1600 m3/s. This range is the result of the random nature of
Monte Carlo simulations and is unacceptably large.
If importance sampling is applied, the variance in estimated peak discharges decreases dramatically
and estimates are available for a much wider range of probabilities. At the same time, a lower number
of simulation runs is required, thus saving computation time. Furthermore, a comparison with the
benchmark (green dots) shows no bias is introduced by the importance sampling. This clearly
demonstrates the potential advantages of importance sampling.
annual exceedance probability (1 in N)
discharge(m3/s)
10
0
10
1
10
2
10
3
10
4
10
5
10
6
0
200
400
600
800
1000
1200
1400
1600
1800
2000
crude MCS; 100 runs; 20,000 samples per run
importance sampling; 100 runs; 5,000 samples per run
crude MCS; 1 run; 100 million samples
Figure 2 Annual exceedance probability and corresponding peak discharges as estimated from
three sets of Monte Carlo runs different (see legend for the details).
5. DISCUSSION
The benefits of importance sampling with regard to increased accuracy and reduced computation time
depend on the specific case to which it is applied. It is particularly beneficial for cases were events
with low annual exceedance probabilities need to be quantified. Optimal sampling strategies will be
different for each case. The strategy presented in the current paper is likely to be efficient in
catchments where the rainfall depth is the dominant factor on the considered outputs, which is often
the case in hydrologic assessments. The strategy is particularly efficient if the scattered relation
between rainfall depth and peak discharge shows a more or less linear trend. If the trend is non-linear,
the efficiency of the method can be increased by adapting the importance sampling function of
equation (10) accordingly. This can be done with relatively minor effort, as long as catchment
response characteristics are known. For catchments were rainfall duration is an additional relevant
factor, it may be worthwhile to apply importance sampling on rainfall durations as well, to increase the
proportion of long duration events. This is most likely beneficial for larger catchments and/or
catchments with significant (reservoir) storage. Adaptive sampling schemes (e.g. Dawson and Hall,
2006; Steenackers et al, 2009) can be used as an alternative to pre-defined sampling strategies. In
such approaches, the importance sampling function is continuously updated, based on the

intermediate simulation results. The advantage is that these methods are able to ‘learn’ from the
simulation results, and therefore do not require expert input on sampling strategies. A disadvantage is
that it is a significant challenge to implement these methods in such a way that they are robust.
6. CONCLUSIONS
Standard (‘crude’) Monte Carlo Sampling schemes, such as the CRC-CH method suffer from a large
variance in design flood estimates for large to extreme events, thus making the method unreliable for
these events. It was demonstrated in the current paper that importance sampling can overcome this
problem, often in combination with a decrease in computation time. This creates opportunities for
using the CRC-CH method in applications where probabilities of extremes are relevant, such as in
dam design studies.
7. ACKNOWLEDGMENTS
The authors want to express their gratitude for the valuable comments on the work described in this
paper by representatives of the Queensland Government, by members of the Technical Working
Group and by members of the Independent Panel of Experts, all of whom are involved in the Brisbane
River Catchment Flood Study.
8. REFERENCES
ARR (1987), Australian Rainfall and Runoff – A guide to flood estimation. Institution of Engineers
Australia
ARR (2013), Australian Rainfall and Runoff – Monte Carlo simulation techniques, Australian rainfall
and runoff discussion paper, May 2013
Carroll D.G. (2012), URBS (Unified River Basin Simulator) V 5.00, Dec 2012
Carroll, D. and Rahman, A. (2004). Investigation of sub-tropical rainfall characteristics for use in the
joint probability approach to design flood estimation. Proceedings of the 2nd APHW
Conference, Singapore, 5-8 July, 2004.
Dawson R and Hal J (2006), Adaptive importance sampling for risk analysis of complex infrastructure
systems. Proc. R. Soc. A, 462 (2075): 3343-3362.
Diermanse, F.L.M., Carroll, D.G., Beckers, J.V.L., Ayre, R. and Schuurmans, J.M. (2014), A Monte
Carlo Framework for the Brisbane River Catchment Flood Study, proceedings of the
HWRS2014 conference in Perth, Australia.
Hoang TMT (2001). Joint probability approach to design flood estimation, Unpublished PhD Thesis,
Department of Civil Engineering, Monash University, Victoria, Australia.
Ilahee, M., Rahman, A. and Boughton, W. C. (2001), Probability-distributed initial losses for flood
estimation in Queensland, Proceedings of the International Congress on Modelling and
Simulation.
Mirfenderesk, H, Carroll, D., Chong, E., Rahman, M., Kabir, M., Van Doorn, R., and Vis, S. (2013),
Comparison between Design Event and Joint Probability Hydrological Modelling. Paper
presented to the Floodplain Management Association national conference 2013, Tweed
Heads, NSW, Australia.
QFC, (2012): Queensland Floods Commission of Inquiry Final Report, 2012,
http://www.floodcommission.qld.gov.au/__data/assets/pdf_file/0007/11698/QFCI-Final-Report-
March-2012.pdf.
Rahman A., Weinmann, P.E., Hoang, T.M.T. and Laurenson, E.M. (2002), Monte Carlo Simulation of
Flood Frequency Curves from Rainfall, Journal of Hydrology 256,196-210.
Sadowski, J. S. and Bucklew, J. A. (1990). On large deviations theory and asymptotically efficient
Monte Carlo estimation. IEEE Trans. Inform. Theory 36, 579-588.
Siegmund, D. (1976), Importance sampling in the Monte Carlo study of sequential tests. Ann. Statist.,
4:673–684.
Steenackers GF, Presezniak F, Guillaume P (2009) Development of an adaptive response surface
method for optimization of computation-intensive models, Comput Ind Eng 57, 847–855.

HWRS2015_Diermanse_Paper_51-Final

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to HWRS2015_Diermanse_Paper_51-Final

Similar to HWRS2015_Diermanse_Paper_51-Final (20)

HWRS2015_Diermanse_Paper_51-Final