Eesti Panga teaduspreemia 2018. aasta laureaatide võidutööde tutvustus

1. PUBLIC AGENCIES’ PERFORMANCE
BENCHMARKING IN THE CASE OF DEMAND
UNCERTAINTY WITH AN APPLICATION TO
ESTONIAN, FINNISH AND SWEDISH FIRE
AND RESCUE SERVICES
Presentation of the PhD thesis at the Bank of Estonia
Tarmo Puolokainen
University of Tartu
November 28, 2018

2. Topic and novelties of the study
Nowadays, one could talk about a practical public sector
management and administration field, which focuses on
improving the management decisions in the public sector. And
a theoretical microeconomic production theory, aiming to
framework the choices and decisions a producer faces. More
often than not, these two fields of study live separate lives.
This thesis tries to merge these fields on the approach to
demand uncertainty in a public agencies’ performance.
The thesis introduces the concept of demand uncertainty into
the public agencies’ performance measurement and
management through frontier analysis methods.

3. Topic and novelties of the study continued
In addition, although quite vast trade press covers the topics
of fire and rescue services, it is a novel and understudied
application in the efficiency studies.
It is the first attempt to analyse systematically the
performance of fire and rescue services in three countries,
Estonia, Finland and Sweden. This opens opportunities for a
discussion on how to reach the levels of Scandinavian fire and
rescue services quality, which is the goal of Estonian Rescue
Board, stated in its strategy.

4. The aim of the doctoral thesis is...
...to develop the theoretical concept and application to measure
the performance of public agencies in the case of demand
uncertainty. The suggested models would be the basis for planning
resource allocation improvement in public agencies. The models
are implemented using the example of the Estonian, Finnish and
Swedish fire and rescue services.

5. Research tasks
Application:
Illustrate with a real
situation in fire and
rescue services
Compare different
methods
Empirically measure
the performance of FRS
Assess possibilities of
using such framework
in decision-making
Methods:
Methods implemented on the measurement of the
public agencies performance
Measurement methodology that would be able to
incorporate demand uncertainty
Theoretical framework:
Defining and measuring the
performance in public agencies
Potential uses of performance
measurements in public agencies
Impact of demand uncertainty
on the public agencies in the
provision of services
Figure: Research tasks (Source: Author’s compilation)

6. Performance, reasons of its variation and intended
purposes of measuring it in the public sector
Performance measurement is the process of quantifying
action, where measurement is the process of quantification
and action leads to performance (Neely et al., 1995, pg 1228).
The performance might differ because agencies’ have different
objectives, needs, ways of service provision, interactions with
other organisations, efficiency, accounting, reporting and
measuring methodology as well as random fluctuations.
So, it might be useful to conduct a performance measurement
in order to plan and improve the work of public agencies or to
evaluate and benchmark against other public agencies.
Planning and improving is of most value for internal purposes;
evaluation and benchmarking, on the other hand, is mostly
targeted for external audiences, who, in turn can influence the
operations of subunits.

7. The dimensions of performance
objectives inputs activities outputs
final
outcomes
intermediate
outcomes
Public agency
needs
environment
socio-econom ic situation
utility and sustainability
relevance
econom y
effectiveness
efficiency
cost effectiveness
other
stakeholders
Figure: The conceptional model of performance (based on Pollitt and
Bouckaert, 2011; Van Dooren et al., 2015).

8. The case of fire and rescue services
Figure: The main levers to reach the goal of a rescue service (based on
Jaldell, 2002).

9. Production theory as a basis of performance measurement
Let’s consider a service industry in which it is not possible to
provide more services than are demanded (cannot store the
outputs, e.g., fire and rescue services).
A ‘central agency’ allocates resources to ‘subunits’ in different
jurisdictions in the face of uncertainty about the services that
will be demanded in each jurisdiction.
The central agency tries to minimise the cost of providing
enough resources to meet a minimum service level (MSL) in
each jurisdiction.
Each subunit tries to use his/her allocated inputs to provide
the services demanded in his/her jurisdiction.
One would be interested in (a) the cost efficiency of the
central agency (b) any under-resourcing of subunits, and (c)
the technical and mix efficiency of each subunit.

10. Demand uncertainty
...has not been addressed on most occasions when assessing
the (cost-)efficiency of public agencies.
And probably have resulted in underestimating their efficiency
in many cases.
Risk averse behaviour of the decision-maker (public agencies
managers face social pressure to satisfy a large percentage of
demand).
The excess capacity is an insurance or more generally a service
to the public, should someone unpredictably require the
services.
The agencies will allocate resources which are ex ante
optimal, given expected demand, but are not ex post efficient,
given realized levels of demand.
The challenge is to distinguish the necessary standby capacity
from excessive mismanagement.

11. MSL
Minimum service level (MSL) - a target level, which would insure
the decision-maker against the upsurges of demand. Such standby
capacity should be incorporated to the analysis, as it alternates the
decision-makers behaviour.
The MSL in a given jurisdiction is the P-th percentile of the
(estimated) probability distribution of services demanded in
that jurisdiction.
Can be chosen freely.
Can be used for contracting purposes when outsourcing the
services.

12. Figure: The conceptional model of demand uncertainty (Source: Author’s
compilation)

13. Formalisation I
Under weak regularity assumptions, the production
possibilities set can be represented using distance functions.
The central agency chooses inputs to minimise the cost of
meeting the MSL in each jurisdiction. The period-t
optimisation problem of the central agency is
min
x≥0
wtx : Dt
I (xi , mit, zit) ≥ 1 for i = 1, . . . , I} (1)
where wt = (w1t, . . . , wIt) and x = (x1, . . . , xI ). The input
vector that solves this problem is x∗
t = (x∗
1t , . . . , x∗
It ).
From there, cost efficiency CEt(xt, wt, mt, zt) = wtx∗
t /wtxt

14. Formalisation II
Subunit i seeks to use the allocated inputs to provide the
services demanded. Subunits’ period-t optimisation problem is
max
q
{Q(q) : q ≤ dit, Dt
O(xit, q, zit) ≤ 1} (2)
where Q(.) is a nonnegative, nondecreasing, linearly-
-homogeneous, scalar-valued aggregator function with weights
that represent the values the subunit places on outputs. The
output vector that solves this problem is ˆqit ≡ ˆqt(xit, dit, zit).
The associated aggregate output is Q(ˆqit).
The output-oriented technical and mix efficiency (OTME) of
subunit i in period t is
OTMEt(xit, qit, dit, zit) = Q(qit)/Q(ˆqit).

15. Possible methods for performance measurement
Figure: Different measurement methods used to measure performance in
a public agency (Source: Author’s compilation).

16. Frontier analysis methods and the current case
The frontier analysis methods, which are mainly used in the
framework of productivity analysis, have a common purpose of
modelling the frontier of feasible performance. The frontier
can be estimated under various underlying assumptions and
estimation methods. As the next step, observed organisation’s
performance indicator is then compared to such frontier and
relative efficiency is found.
Mainly two schools of thought are distinguished: some prefer
econometric methods which use stochastic and parametric
models (e.g Battese and Coelli, 1993; Kumbhakar and Lovell,
2003; Greene, 2005), and others who prefer linear
programming methods which use mainly deterministic and
non-parametric models (e.g Simar and Wilson, 1998;
Thanassoulis, 2001; Cooper et al., 2011).

17. Critique on the frontier analysis methods
Although popular amongst scholars, they are rarely used as
direct policy tools. Such scarce use can be attributed to the
limits of these techniques (Daraio and Simar, 2007).
The impossibility to extend the efficiency analysis beyond the
current regression sample - making it useless for subunits not
included in the sample. Similarly, comparison of two different
efficiency studies is of little use.
Several assumptions made about the production function and
inefficiencies cannot be successfully verified, and ensuring the
robustness of the results is complicated. Similarly, the lack of
knowledge about the ‘true’ production process restricts the
development of convincing theoretical model (Martin and
Smith, 2005).
Initial result of efficiency analysis is a single composite
measure, which might not be helpful from managerial point of
view.
There is no consensus on how to take the environmental
influences and dynamic effects into account.

18. ESTONIAN, FINNISH AND SWEDISH FIRE AND
RESCUE SERVICES
Estonian FRS is centralized and managed by Estonian Rescue
Board (since 2012). Altogether 72 national FRS brigades and
115 voluntary FRS brigades (not included into the analysis
due to data limitations). Analysis of Estonian FRS is done on
FRS brigade level.
Finnish FRS is more decentralized and organised
independently by 22 fire departments, under which are 370
national, 523 contracted (voluntary), and 105 industrial FRS
brigades. The analysis is done on fire department level, which
is the smallest possible unit with available data.
Swedish FRS is even more decentralized - the FRS are offered
by municipalities (290), which co-operate (165 FRS
authorities). The analysis is done on municipality-level. The
data is unbalanced.

19. Notation
I = 65 brigades / 22 fire departments / 126 − 275
municipalities
T = 5 periods / 12 periods / 11 periods (-2015)
q1it = fires in buildings
q2it = other fires
q3it = traffic accidents
q4it = other emergencies
x1it = labour (no. of employees)
x2it = other inputs (assumed proportional to no. of
vehicles/no. of FRS brigades)
zit = 1/area (because harder to service large areas)
dnit = qnit (i.e., all demands for service were met)
mnit = the value such that Pr(dnit ≥ mnit) = 0.05

21. Figure: Number of emergency departures per 100,000 population in
Estonia, Finland and Sweden (Source: Estonian Rescue Board 2016;
PRONTO 2016; IDA 2016; Author’s calculations)

22. Figure: Costs (’000 of 2011 e) of FRS per 100,000 population in Estonia,
Finland and Sweden (Source: Estonian Rescue Board 2016; PRONTO
2016; IDA 2016; Author’s calculations)

23. Industry-level cost-efficiency of FRS
The cost-efficiency has been calculated for each country using
a na¨ıve model and a model, that accounts for the uncertain
demand using minimum service level - MSL.
Different estimation methods were used - data envelopment
analysis (DEA), free disposal hull (FDH) and deterministic
frontier analysis (DFA).

24. The cost-efficiency and potential savings (thousands of
2011 euros) of ERB using the DEA
Real Na¨ıve Na¨ıve
Year costs CE CE savings Savings
(’000 e) (0,1] (0,...) (’000 e) (’000 e)
2011 22, 776 0.838 1.058 3, 698 −1, 011
2012 28, 111 0.843 0.902 4, 411 2, 324
2013 25, 857 0.753 0.930 6, 389 1, 518
2014 23, 820 0.814 0.919 4, 438 1, 650
2015 36, 304 0.772 0.906 8, 291 2, 723
Na¨ıve CE obtained by replacing mit with qit.
Potential savings are in thousands of 2011 Euros.

25. The cost-efficiency and potential savings (thousands of
2011 euros) of ERB using the FDH
Real Na¨ıve Na¨ıve
Year costs CE CE savings Savings
(’000 e) (0,1] (0,...) (’000 e) (’000 e)
2011 22, 776 0.985 1.781 334 −13, 200
2012 28, 111 0.976 1.626 670 −13, 163
2013 25, 857 0.962 1.634 980 −12, 037
2014 23, 820 0.975 1.411 587 −7, 397
2015 36, 304 0.956 1.470 1, 606 −11, 529
Na¨ıve CE obtained by replacing mit with qit.
Potential savings are in thousands of 2011 Euros.

26. The cost-efficiency and potential savings (thousands of
2011 euros) of ERB using the DFA
Real Na¨ıve Na¨ıve
Year costs CE CE savings Savings
(’000 e) (0,1] (0,...) (’000 e) (’000 e)
2011 22, 776 0.491 0.529 11, 601 10, 737
2012 28, 111 0.476 0.530 14, 743 13, 203
2013 25, 857 0.418 0.511 15, 045 12, 638
2014 23, 820 0.489 0.549 12, 168 10, 737
2015 36, 304 0.360 0.411 23, 243 21, 372
Na¨ıve CE obtained by replacing mit with qit.
Potential savings are in thousands of 2011 Euros.

27. Figure: The Pearson correlations, densities and scatterplots of
cost-efficiency estimates using different methods (Source: Estonian
Rescue Board, Authors’ calculations).

28. Insights
Estonia
CE has decreased in time - the number of emergencies decreased,
the costs increased.
CE estimates correlate weakly negatively with the population
reached within 15 minutes. So, the FRS brigades with less
population in close vicinity would be estimated higher CE.
CE estimates correlate weakly positively with the average arrival
time to the scene. One can argue, that being faster is costlier.

29. Under-resourcing of FRS subunits
After the central agency (or government/municipality) has
allocated the resources between different subunits, the subunits
have to respond to emergencies with the given input bundles (a
fixed number of rescuers and vehicles).
The standard case - the input-oriented technical efficiency
(ITE) would be calculated in order to analyse, whether the
subunits would have been able to respond to the observed
number of emergencies with fewer amounts of inputs.
Introducing the MSL to the ITE framework, one is able to
distinguish the FRS subunits that would have not been able to
meet the expected MSL, in case the demand would have been
higher as it were observed.

30. Table: The percentage of under-resourced FRS subunits in Estonia,
Finland and Sweden, estimated by DEA, FDH and DFA
DEA FDH DFA
EST FIN SWE EST FIN SWE EST FIN SWE
2004 40.9 40.9 4.5
2005 27.3 14.2 59.1 63.0 0 3.4
2006 22.7 12.7 59.1 61.1 0 4.1
2007 27.3 13.0 59.1 62.2 0 4.3
2008 22.7 12.4 54.5 56.9 0 2.4
2009 27.3 10.2 59.1 55.8 4.5 3.7
2010 18.2 8.4 59.1 54.4 4.5 5.9
2011 61.5 18.2 7.6 87.7 63.6 52.7 16.9 4.5 4.7
2012 36.9 18.2 8.9 87.7 59.1 55.7 16.9 4.5 7.3
2013 33.8 18.2 8.8 87.7 54.5 54.9 16.9 4.5 6.0
2014 32.3 18.2 8.8 80 59.1 54.6 13.8 4.5 6.3
2015 29.2 18.2 7.7 78.4 54.5 55.3 13.8 4.5 10.1
Source: Estonian Rescue Board 2016; PRONTO 2016; IDA 2016;
Author’s calculations.

31. Insights
Although the indications of under-resourced subunits of FRS
are not very robust and alternate between different models, in
most cases the trend has been to better resource allocation
(with an exception of Swedish municipalities estimated by
DFA). This means, that with time, the share of
under-resourced subunits has decreased.
That might be due to fewer emergencies, as the number of
inputs has stayed quite steady across years (outputs decrease
as inputs stay constant). This complies with the assessment
to the cost-efficiency of FRS.

32. Output efficiency of subunits
The central agency is interested in how well the resources are utilized by
the local subunits in different jurisdictions in comparison to their most
efficient counterparts. For that, the output-oriented technical (and mix)
efficiencies (OTME) should be estimated. This indicates, how many more
emergencies the subunits could have responded to, in case there would
have been demand for.
When taking the demand uncertainty into account one can argue, that
the OTME should be one (the FRS subunits are efficient) if they are able
to respond to every emergency in their service area - which is the current
case (demand does not exceed the supply).
While taking this into account, one cannot expect that a FRS subunit
would increase its outputs (as services cannot be stored). In other words,
even if a FRS subunits would have been able to respond to more
emergencies, there was no demand for that (and one should not label this
as inefficiency).
So, only the na¨ıve OTMEs will be estimated, which would demonstrate
the potential of FRS subunits.

33. Figure: The boxplots of estimated OTMEs in Estonia using DEA (Source:
Estonian Rescue Board, authors’ calculations).

34. Figure: The boxplots of estimated OTMEs in Estonia using FDH (Source:
Estonian Rescue Board, authors’ calculations).

35. Figure: The boxplots of estimated OTEs in Estonia using DFA (Source:
Estonian Rescue Board, authors’ calculations).

36. Insights
Estonia
OT(M)Es are consistent in terms of fluctuations between years
in different models, the lowest estimates are in year 2013
(when also least emergencies occurred).
The correlations between the estimates of different methods
are positive (0.48-0.73), with population positive and with the
average time to the scene negative.

37. Discussion
The concept of demand uncertainty is new to efficiency
studies. Such framework of demand uncertainty as a
component in efficiency studies produced plausible results
(when taking into account, the potential savings are lower,
which is an expected result).
MSL as a concept can be appealing for planning activities
(outsourcing, contracting, etc.), also, it can be of importance
for popularising the efficiency studies.
The results are quite robust across methods.
CET-type production function produces plausible results
(theoretically-plausible functional form; coefficients have signs
that are consistent with prior expectation; most are
statistically significant; surprisingly (?) high elasticity of
scale).
Identifying the under-resourced FRS subunits would allow to
improve the resource allocation process.
OT(M)E would indicate the potential of FRS subunits in case
of upsurges in demand.

38. Discussion continued
These models do not account for the whole spectrum of
activities (e.g. prevention), so promoting tunnel-vision.
Limitation of the comparison between countries.
Structure of the management.
Differences in the data sets.
DEA and FDH would get into trouble with the biggest
subunits in the sample, as the MSL would lie outside the
estimated frontier.

39. SUMMARY
The thesis developed a concept for analysing the performance
of public agencies under demand uncertainty.
The framework considers a two-tier decision-making structure
involving a ‘central agency’ and a group of ‘subunits’.
DEA, FDH and DFA can be used to evaluate (a) the cost
efficiency of the central agency, (b) any under-resourcing of
subunits, and (c) the technical and mix efficiency of each
subunit.
In an empirical illustration, one can find evidence for Estonian,
Finnish and Swedish FRS subunits that (a) the cost-efficiency
would be estimated higher when taking the demand
uncertainty into account, and (b) in most estimates, there are
some FRS subunits, who would have not met the targeted
MSL. (c) The potential of FRS subunits to respond to more
emergencies has been identified by all the models.

40. Extensions
The demand uncertainty and the alteration of decision-making
process, effect on performance
MSL can be of interest to analyse the negotiation issues
(different stakeholders have different goals)
Stochastic Frontier Analysis (SFA) framework
Different application
Using the arrival times to the scene as indicators for MSL
Effects of voluntary FRS brigades (Estonian case)
Other outputs (e.g., prevention activities)
Factors affecting MSLs (e.g., prevention activities, choice of
α)
Spillovers (i.e., providing services in another jurisdiction)
Environmental uncertainty (e.g., weather, population)
Productivity analysis

41. Thank you for your attention!
Questions, comments, recommendations.
Tarmo Puolokainen
tarmo.puolokainen@eas.ee

42. Battese, G. E. and Coelli, T. J. (1993). A stochastic frontier
production function incorporating a model for technical
inefficiency effects. Working Papers in Econometrics and Applied
Statistics, 69. Department of Econometrics, University of New
England Armidale.
Cooper, W. W., Seiford, L. M., and Zhu, J. (2011). Data
envelopment analysis: history, models, and interpretations. In
Cooper, W. W., Seiford, L. M., and Zhu, J., editors, Handbook
on data envelopment analysis, pages 1–39. New York, NY:
Springer.
Daraio, C. and Simar, L. (2007). Advanced Robust and
nonparametric methods in efficiency analysis: methodology and
applications. New York, NY: Springer Science & Business Media.
Estonian Rescue Board (2016). A combined database from internal
sources. A statistics database gathered especially for the thesis
in cooperation with the Estonian Rescue Board.

43. Greene, W. (2005). Fixed and random effects in stochastic frontier
models. Journal of productivity analysis, 23(1):7–32.
IDA (2016). Statistics system of Swedish rescue services (IDA). A
statistics system developed and maintained by the Swedish Civil
Contingencies Agency [https://ida.msb.se/].
Jaldell, H. (2002). Essays on the performance of fire and rescue
services. PhD thesis, G¨oteborg University,.
Kumbhakar, S. C. and Lovell, C. K. (2003). Stochastic frontier
analysis. Cambridge: Cambridge University Press.
Martin, S. and Smith, P. C. (2005). Multiple public service
performance indicators: Toward an integrated statistical
approach. Journal of Public Administration Research and
Theory, 15(4):599–613.
Neely, A., Gregory, M., and Platts, K. (1995). Performance
measurement system design: a literature review and research
agenda. International Journal of Operations & Production
Management, 15(4):80–116.

44. Pollitt, C. and Bouckaert, G. (2011). Public Management Reform:
A comparative analysis-new public management, governance,
and the Neo-Weberian state. Oxford: Oxford University Press.
PRONTO (2016). Statistics system of Finnish rescue services
(PRONTO). A statistics system developed and maintained by
the Emergency Services College in Kuopio, Finland
[https://prontonet.fi/].
Simar, L. and Wilson, P. W. (1998). Sensitivity analysis of
efficiency scores: How to bootstrap in nonparametric frontier
models. Management Science, 44(1):49–61.
Thanassoulis, E. (2001). Introduction to the theory and
application of data envelopment analysis. Boston, MA: Springer.
Van Dooren, W., Bouckaert, G., and Halligan, J. (2015).
Performance management in the public sector. London:
Routledge.