The document outlines a research project aimed at developing a framework for designing and evaluating command post exercises in disaster response organizations. It discusses challenges encountered during exercises, the need for defined performance criteria, and the importance of an organized evaluation methodology. The research culminates in a guide to support practitioners in achieving effective and goal-oriented exercise designs.

1. Training, Testing and Experimentation:
A classification of command post exercises
Action research approach with staffs of disaster
response organizations
Erich Heumüller
Sebastian Richter
Ulrike Lechner

2. Research Design
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 2
Goal of
research
Development of a framework for a goal-oriented and evaluation-driven exercise design
Research
Partners
THW (Federal Agency of Technical Relief): coordination and communication units
Bundeswehr: Kreisverbindungskommando Fürth (liaison unit at the county level)
Research
Context
• Our goal is to support practitioners in exercise design (handbook or guideline)
• Involvement in four command post exercises
• We follow an Action Research approach, which aims to solve a practical problem and to
generate scientific knowledge

3. Research project
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 3
Framework for Exercise Design
Heumüller, Richter, Lechner: ISCRAM 2012
Conceptual Model of Staffs
Heumüller, Richter, Lechner: eBled 2012
Classification of
command post exercises
Heumüller, Richter, Lechner:
ISCRAM 2013

4. Action Research Design
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 4
4 - 6 different staffs per exercise
THW staff: 7 up to 15 members
Reservists’ staff: 3 to 6 members
The exercise classification approach was an “Action Planning”
result of Cycle 1 and applied in the following research

5. Challenges of staff exercising
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 5
A valid staff assessment is difficult,
because performance criteria are not
defined:
„What is a good staff?“
After the first exercise it was hard to
say, which staff performed best.

6. Diagnosing
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 6
Data gathering Unit of Analysis Exemplary Findings
Literature
Analysis
Evaluation Reports like
“Wintersturm” (2007),
“Burgstadt unter
Wasser” (2008)
•No information about e.g.
Participants’ training level,
Special processes observed,
Lack in resources,
Structural or technical shortcomings or other findings,
Exercise goal or purpose.
Involvement in
Command Post
Exercises
GERETSRIED •No evaluation criteria for a shift change
•No test results
•No information about the “unknown” process
•Participants were frustrated
GROSSER KREIS •Team missed guidance for exercise conceptualization,
although regulations for exercise design were known
•Team puzzled about goal, topic and purpose
•Team lost in detail during inject development
•Almost at the end of planning suddenly a bunch of
evaluation preparation started
•Observers were not consulted, methods were unknown
•A known traffic light system was perceived to be useless
for evaluation

7. Action Planning - State of the Art of Exercise Classifications
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 7
Exemplary
Authors /
Articles
- Emergency
Management
Division - Michigan
State Police, 1998,
- Oregon, 1997,
- Virginia
Department of
Emergency
Management,
2001
-US Dept of
Homeland
Security, 2007
- Renner, 2001 - Callan, 2009,
- Australian
Emergency
Management
Institute, 2012;
- Australian
Government, 2007
- Payne, 1999 -United States Federal Emergency
Management Agency, 2012,
-Trnka and Jenvald, 2006,
-Klein et al., 2005,
-Perry, 2004,
-Peterson and Perry, 1999,
-Ohio Emergency Management Agency, 1999,
-Texas Department of Public Safety, Division
of Emergency Management, 2000
Exercise types
Seminar Orientation Exercise Seminar
Information
Session
Seminar, Agency
Presentation
Workshop Workshop Workshop
Walkaround Walkaround
Paper-Feed
Exercise
Paper-feed Exercise
Tabletop
Exercise
Tabletop Exercise Tabletop Exercise
Tabletop
Exercise
Syndicate,
Hypothetical Exercise
Table-top Exercise Tabletop Exercise
Game Game
Drill Drill Drill
Functional
Exercise
Functional Exercise Functional Exercise
Centre
Simulation
Functional Exercise
Communications
Simulated Exercise
Functional Exercise
Full-Scale
Exercise
Full-Scale Exercise Full-Scale Exercise Field Exercise Field Exercise Live Exercise Full-Scale Exercise

8. Training, Testing and Experimentation:
A classification of command post exercises
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 8

9. Action Taking – Our classification approach in application
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 9
Identified changes:
Exercise
Organiz
ation
Date Changes
FEUERBALL THW 7th-9th,
October,
2011
•First planning team discussed uncoordinated about ideas, injects,
exercise goal
•We concentrated on our approach  exercise design was defined
well
•Evaluation was a central issue  deduction of criteria from
regulations and specifications for injects
•Injects served to test management skills and not to produce stress
•Evaluation guidelines for observers were prepared, based on
identified evaluation criteria
•Evaluation report offers valid statements about participants’ training
level
FRANKEN-
STURM
THW 5th-7th
October,
2012
•Goal was well defined and documented very early  identification of
something new ( a novelty in THW exercises)
•Scenario was not conceptualized to produce stress, but to support
evaluation
•For the first time, exercise course followed evaluation methodology:
the core was a before-and-after measurement of identified criteria to
analyze the shift change
•New processes were identified  baseline for the development of
new regulations

10. Evaluation and Learning
• Our approach supported common understanding and served as a first guideline for
exercise design.
• The terms and definitions Training, Test and Experimental Exercise were adopted
quickly.
• Our approach provides a frame for the planning team to develop the exercise content
and to prepare the evaluation.
• The planning team of FEUERBALL recommended our approach to the planning team of
FRANKENSTURM. This team perceived a high value using this approach.
• Although our approach was perceived to be logically and comprehensible, there is still
professional guidance required to apply this approach conclusively.
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 10

11. Discussion
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 11
Empirical base is limited to four
command post exercises; classification
was applied in only two of them.

12. Next Steps
What was achieved?
• Development of a framework which provides practical guidance for goal-oriented and
evaluation-driven exercise design for different exercise types.
• Development of a conceptual model of disaster response staffs, which supports
evaluation of exercises.
What is to do?
• Involvement in this year’s exercise in MELLRICHSTADT (northern Bavaria).
• Focus will be on a link between the staff model and the framework to enable a goal-
oriented exercising of staff skills.
• Practical adaption of our templates regarding usability, notations and descriptions.
24.06.2013 A classification of command post exercises - © Heumüller, Richter, Lechner 2013 12