CSSI’s Kim Bender was a speaker at 2014's AMS Summer Community Meeting: Improving Forecasts and the Communication of Forecasts. Kim was a member of the panel on “Synthesizing Forecasting Information” which discussed the plethora of information forecasters have to guide their decisions.

1. Optimizing Data Synthesis
and Visualization in Real-
Time Decision-Making
Kim Bender, AMS Summer Meeting
August 2014

2. “As weather forecaster you will be asked to make decisions
in situations where you will be under tight time constraints
and face uncertainty, you will have huge amounts of data
available, some of it contradictory and most of it not really
of significance to you, and you will be constantly looking for
the correct data, which will not normally be at your disposal
(Gaia & Fontannaz, 2006, p.17)”
Confidential and Proprietary. Do Not Distribute. 2

3. Data Synthesis and Visualization in Real-Time
Decision Making
• Forecasting and aviation operations are both technological environments
where data is increasingly being incorporated into operator workstations to
optimize accuracy and efficiency
○ Automation
○ Decision Support Tools
○ Systems
Confidential and Proprietary. Do Not Distribute. 3
• Issues and questions:
○ How do you ensure the right systems are incorporated?
○ How do you ensure effective presentation and synthesis of information?
○ What are the impacts of the system on the human operator, safety, and
operations?
• What predisposes these operations to these environments?
Operators are required to make real-time decisions
in dynamically changing, uncertain environments

4. Human Factors: Decision Making in Uncertainty
• Decisions Making in Uncertainty
○ Heuristics – intuitive processes based on individual experience and perceptions
○ Objective data – structured and unstructured data but usually statistical in nature
• Heuristics mandatory in uncertain environments: Intuition and experience help
humans make quick effective decisions
○ Best human forecasters can consistently outperform objective methods-
Confidential and Proprietary. Do Not Distribute. 4
Heuristics!
○ Heuristics influenced by past experience and subjective considerations which can
be positive or negative
■ Positive – recognize similar situation or pattern and apply lessons learned when data is
uncertain
■ Negative – can introduce bias due to previous negative outcome or experience or it
can lead to missed event
□ Previous false alarm, forecasters tends to under forecast next event leading to a missed
alarm
□ Previous missed alarm, forecasters tend to over forecast after missed event

5. Human Factors: Decision Making in Uncertainty
• Inordinate amounts of objective data for real-time decision-making can be
problematic when not introduced effectively
• Operator must mine/culls/filter data to quickly synthesize
○ Multiple data sources can introduce more uncertainty
■ Contradictory output consistencies and forecasts
■ Individual data interpolation and integration/translation differences
○ Data overload – huge amounts of data
■ Contradictory data sources from disparate sources and systems
■ Most data not really of significance for task at hand
Confidential and Proprietary. Do Not Distribute. 5
○ Data Visualization
■ Scanning for correct/relevant data can be time consuming and problematic
■ Relevant data may not always be available or clear
■ Multiple systems introduce display and output inconsistencies
If data synthesis and visualization can be structured
carefully, could mitigate many significant issues

6. Evaluating Effective Data Presentation and
Visualization
• FAA tests proposed advanced aviation systems, procedures, and concepts
and/or performs experimental research to test proposed change
○ Subject Matter Expert (SME) Panels
Confidential and Proprietary. Do Not Distribute. 6
○ Simulation Research
○ Field Research
• Introduction of new data, information, and/or display in aviation environment
requires:
○ Operational Testing and Evaluation using SME’s
○ Measurement/evaluation of impacts of proposed technologies, displays, or
concepts on operations, safety, or human operator
○ SME evaluation of information display

7. Key Considerations for Data Synthesis and
Visualization
• Data introduction and visualization considerations
○ Data selection, design, display and is key to performance
○ Only present data that is meaningful and accurate - “too much” data will
eventually decrease performance by adding ineffective workload
• Implement a systematic approach to data presentation and visualization
○ SME and user evaluations during development and operational testing
○ Disciplined and systematic approach data presentation and visualization display
characteristics / system integration
■ Keep systems visually consistent – colors, presentation, warnings, etc.
■ System integration –data consistency
Confidential and Proprietary. Do Not Distribute. 7

8. Key Considerations for Data Synthesis and
Visualization
• Experimental research to understand the effects of the system on human
performance and evaluate alternative approaches
• Understand human decision making and how operator
visualizes/interacts with large datasets in operational settings
○ Understand each operation – all different
○ Cognitive walkthroughs and walk-through like methods
• What data should be presented and how much to automate?
○ Some processes/tasks should not be automated
○ Keep human-in-the-loop to ensure they are active, involved, and aware
○ Research and study need to determine best automation candidates
○ Operational design needs to be structured to tasks
Confidential and Proprietary. Do Not Distribute. 8

9. Key Considerations for Data Synthesis and
Visualization
• Workstation should make best use of human strengths:
○ Recognizing patterns
○ Using conceptual models and formulating mental models
○ Judgment and decision making when dealing with complex, incomplete or
Confidential and Proprietary. Do Not Distribute. 9
conflicting data
○ Applying adaptive strategies in rapidly changing situations
• Human prefers ability to align data to their mental model
○ Too much data can disconnect human from process
○ Presentation of data can weaken mental models
■ Adaptability of display preferences and visualization of data
■ Human control over decision support tools is key - adapt/accept/reject decisions

10. Key Considerations for Data Synthesis and
Visualization
• New outputs or systems can be ignored or dismissed
○ Add workload under time constraints
○ Time and training to learn system may be inadequate
• Humans must trust and understand system for it to influence “decision”
• Style of forecaster influences use of tool
Confidential and Proprietary. Do Not Distribute. 10

11. Forecasting Data Synthesis Considerations
• Understand forecasting decision-making better to formulate tools, mitigate
data overload and ensure optimal visualization/ interaction with systems and
workstations
○ Little work has been done to date specific to forecasting community
○ Differentiate needs between forecasting environments (i.e. severe weather
forecasting; nominal vs off nominal operations)
Confidential and Proprietary. Do Not Distribute. 11
• Heuristics Enhancement
○ Implement “lessons learned” approach to identify potential biases or possible
inclination for missed alarm/false alarm
○ Implement collaborative decision making approach to reduce introduction of
individual previous experience bias
• Study best forecasters and poor forecasters to determine what influences
accuracy and understand performance indicators

12. References
Doswell III, D. A. (2004). Weather forecasting by humans - heuristics and decision making.
Journal of Weather and Forecasting, 19, 1115-1126.
Gaia, M., & Fontannaz, L. (2006). The human side of weather forecasting. The European
Confidential and Proprietary. Do Not Distribute. 12
Forecaster, 14, 17-20.
Kroonenberg, F. (2000). Human Factor in Severe Weather Forecasting. Paper presented at 9th
European Conference on Applications of Meteorology/9th European Meteorological
Society (EMS) Annual Meeting, 28 September – 02 October 2009, Toulouse, France,
13-10-2009. Retrieved July1, 2014, from
http://www.emetsoc.org/fileadmin/ems/dokumente/annual_meetings/2009/AM1_EMS20
09-624.pdf
Sills, D. M. (2009). On the MSC forecasters forums and the future role of the human forecaster.
Bulletin of the American Meteorological Society, May 2009, 619-627.
Stuart, M. A., Schultz, D. M., & Klein, G. (2007). Maintaining the role of humans in the forecast
process: Analyzing the psyche of expert forecasters. Bulletin of the American
Meteorological Society, December 2007, 1893-1898.