The increasing dynamic and complexity of todays global supply chains and the growing amount and complexity of information challenge deci- sion makers in manufacturing companies. Decision Support Systems (DSS) can be a viable solution to address these challenges and increase the overall decision efficiency and effectivity. However, a thought-through design and implementation of these systems is crucial for their efficacy. This article presents the current state-of-the-art of Decisions Support Systems and highlights their benefits and pitfalls. Also, we present an empirical study in which we compared different levels of decision support and decision automation in a simulated supply chain game environment. We identify and quantify how human factors influence the decision quality and decision performance in this supply chain scenario. We show that an adequately designed system raises the overall performance. However, insufficiently designed systems have the reverse effect and lead operators to miss severe situations, which can have fatal consequences for manufacturing companies.

1. How Correctness of Decision Support
Systems Influences User’s Performance
in Production Environments
Defective Still Deflective
Philipp Brauner
André Calero Valdez
Ralf Philipsen
Martina Ziefle
Human-Computer Interaction Center
RWTH Aachen University, Germany
Human-Computer Interaction
International 2016
Toronto, Canada
Philipp Brauner , André Calero Valdez, Ralf Philipsen, Martina Ziefle, Defective Still
Deflective – How Correctness of Decision Support Systems Influences User’s Performance
in Production Environments, HCI in Business, Government, and Organizations: Information
SystemsVolume 9752 of the series Lecture Notes in Computer Science pp 16-27, 978-3-
319-39398-8, Springer International Publishing (2016)

2. RWTH Aachen University, Human-Computer Interaction Center
Context: Part of the Cluster of Excellence
“Integrative Production Technology for High-Wage Countries”
 Goal: Strengthen competitiveness of high wage countries
 Engineering of future production systems
– New materials and processes
– Improved and smarter machinery
– Optimize assembly cells, shop floor, cross-company cooperation
 > 25 Institutes, > 100 researchers, > 60M€
page 2
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016

3. RWTH Aachen University, Human-Computer Interaction Center
Our research objective:
Optimize cross-company cooperation
 Optimize cross-company supply chains (SC)
– Technical factors influencing performance of SCs
– Human Factors influencing performance of SCs
– Interface Factors on SCs performance
– Interrelationship of technical, interface, and human factors
 Why are humans considered?
– Humans make final decision
– Overview over not explicitly modelled relationships
(e.g., closed-world assumption)
– Complexity and uncertainty increases,
less time for making decisions
Information flow
flow of goods
Supply Chain
page 3
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
Goal: Understand system and user factors that influence efficiency, effectivity,
and user satisfaction in Enterprise Resource Planning Systems, Supply Chains
and Quality Management.

4. RWTH Aachen University, Human-Computer Interaction Center
What determines performance in
Complex Cyber-Physical Production Systems?
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 4
Domain expertise
Personality states & traits
Trust, Self-efficacy,
Motivation, …
Uncertainty, randomness
Non-linear interactions
Disruptions & Seasonal changes
Feedback loops, …
Interface Design
Visual complexity
Information visualization
Decision Support, …
Efficiency, Effectivity,
Profit, Quality, Satisfaction
Of Workers and Customers
USERSYSTEM INTERFACE
PERFORMANCE


5. RWTH Aachen University, Human-Computer Interaction Center
How can Human and Interface factors be investigated?
Business Simulation Games!
 Convergence between field and laboratory study
 Simplified & controllable (game-based) environments
 Experimentally manipulate complexity and interface
 Empirical methodology to quantify human performance
– Identify and measure influencing personality factors
– Identify and measure influencing interface factors
– Build a formal model that explains performance
 Side-effect:
Usable for game-based learning (GBL) in
education and professional trainings
Test in the field
(ecological validity)
Controlled
experiment in
laboratory
(internal validity)
We are here
page 5
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016

6. RWTH Aachen University, Human-Computer Interaction Center
Business Simulation Games as a
Research Lab for Understanding System, Interface, and User Factors
 Interactive Business simulations
– Forrester’s Beer Distribution Game, Goldratt’s Game
– Quality Management Game
 Several studies
– Do System, Interface, and Human factors influence
performance?
 Questions addressed
– Replication of similar studies? ✓
– Raises awareness for Quality Management? ✓
– Do human factors exist that explain performance? ✓
– Which human factors influence performance? ❓
– Do interface aspects influence performance? ❓
– Which interface aspects influence performance? ❓
– How can users be supported to make better decisions? ❓
P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
2016-07-20
page 6

7. RWTH Aachen University, Human-Computer Interaction Center
Study with Business Simulation Game
Can Interfaces support Users?
 Research question:
– Do interfaces influence player’s performance?
 Interface optimizations based on user feedback
– Better spatial layout (e.g., process-oriented)
– Key Performance Indicators (e.g., stock level)
 Method
– Study (N=40) with old vs. new interface, surveys
(new interface randomly present in 1st or 2nd round)
 Results
– Users preferred revised user interface
– Higher profits and higher product quality w. new interface
 Conclusion:
– Good interfaces crucial for performance (V = 0.263, F1, 38 = 13.548, p = .001 < .05*)
revised interface
first interface
page 7
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016

8. RWTH Aachen University, Human-Computer Interaction Center
Follow up studies
Focus on singular decisions, focus on specific elements
 Narrow down factors that influence
decision quality and decision speed
 Focus on single decisions in
context of material disposition
 Research Questions
– Which factors explain performance
– Quantify costs of the user interface
– Understand interrelationships between factors
 Here:
Influence of Decision Support Systems (DSS)
Assets Drawbacks
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 8

9. RWTH Aachen University, Human-Computer Interaction Center
Decision Support Systems (DSS)
 “Decision Support Systems aid in solving problems by
automatizing the programmable part of a decision
problem”
[Gorry & Morton 1971]
 Support for operative or strategic tasks
 Support by
– identifying relevant information
– Compile data
– Prepare data
– Visualize data
– Identify actions
– Suggest actions
– Support action execution
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 9

10. RWTH Aachen University, Human-Computer Interaction Center
Research Questions
 Do Decision Support Systems influence
operators’ performance?
– Reaction Times
– Accuracy
 Do operators follow defective Decision Support Systems?
– Reaction Times
– Accuracy
 Does the influence of a Decision Support System relate to the
task complexity?
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 10

11. RWTH Aachen University, Human-Computer Interaction Center
Experimental Setup
 Task in Material Disposition Context
– Compare tables with two columns (Stock level and demand)
– Press key
 Order : If at least in one line a order is necessary (50%)
 No Order : Otherwise (50%)
 Length of the tables (within-subject)
– 2, 6, or 12 lines (short, medium, long)
 3 Decision Support Systems (within-subject)
– None (baseline), Correct DSS, Defective DSS (wrong in 50% of the trials!)
 Measured:
– Reaction Times [ms]
– Accuracy [%]
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 11
Product In Stock Required
Milk 136 98
Sugar 993 124
Flour 245 248
Butter 241 210

12. RWTH Aachen University, Human-Computer Interaction Center
Description of the Sample
 20 participants
 Age
21 – 55 years
29.6±7.2 years
 Gender
8 female, 12 male
 Explanatory variables
– Perceptual speed [n.s.]
– Trust in Automation [n.s.]
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 12

13. RWTH Aachen University, Human-Computer Interaction Center
Results
Baseline experiment (no Decision Support System)
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 13

14. RWTH Aachen University, Human-Computer Interaction Center
Baseline Experiment (no DSS)
Factor Decision
 Significant main effect of the decision
Procurements faster, but less accurate
than non-procurements
 Interpretation:
Linear processing of the presented data.
Search terminated if procurement is
required; search then terminated.
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 14
75%
80%
85%
90%
95%
100%
0
1000
2000
3000
4000
5000
6000
7000
No procurement Procurement
Accuracy[%]
ReactionTimes[ms]
Performance [ms] Accuracy [%]
[V=.870,F2,17=56.883,p<.001,η2=.870]

15. RWTH Aachen University, Human-Computer Interaction Center
Baseline experiment (no DSS):
Factor Table Length
 Significant effect of table length
 Reaction times increase with length of table
 Accuracy decreases with increasing length
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 15
75%
80%
85%
90%
95%
100%
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
2 lines 6 lines 12 lines
Accuracy[%]
ReactionTimes[ms]
Performance [ms] Accuracy [%]
[V=.966,F4,15=100.482,p<.001,η2=.966]

16. RWTH Aachen University, Human-Computer Interaction Center
Results
Effect of the Decision Support Systems
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 16

17. RWTH Aachen University, Human-Computer Interaction Center
Effect of a Decision Support System of Effectivity and Efficiency
 Correct DSS (compared to baseline)
– Reduced Reaction Times
– Increased Accuracy
 Defective DSS (compared to baseline)
– No significant difference
– Trend to reduced reaction times
– Trend to decreased accuracy
⇒ Closer investigation
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 17
75%
80%
85%
90%
95%
100%
0
1000
2000
3000
4000
5000
6000
Correct DSS no DSS
(baseline)
Defective DSS
Accuracy[%]
Performance[ms]
Performance [ms] Accuracy [%]
[V=.681,F4,15=8.006,p<.001,η2=.681]

18. RWTH Aachen University, Human-Computer Interaction Center
Effect of Decision Support Systems on Reaction Times
Closer examining the influence of the Number of Lines
 Correct Decision Support System
– Small effect for low number of lines
– Larger effect for more lines
 Defective Decision Support System
– Small effect for low number of lines
– Larger effect for more lines
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 18
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
Correct DSS no DSS
(baseline)
Defective DSS
ReactionTimes[ms]
2 lines 6 lines 12 lines
[V=.718,F2,17=3.495,p<.001,η2=.718]

19. RWTH Aachen University, Human-Computer Interaction Center
Effect of Decision Support Systems on Accuracy
Closer examining the influence of the Number of Lines
 Correct Decision Support System
– Positive influence on accuracy for all three
conditions
– Highest increase for 12 lines
 Defective Decision Support System
– Negative influence for all three conditions
– Strongest decrease for 12 lines
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 19
75%
80%
85%
90%
95%
100%
Correct DSS no DSS
(baseline)
Defective DSS
Accuracy[%]
2 lines 6 lines 12 lines
[V=.718,F2,17=3.495,p<.001,η2=.718]

20. RWTH Aachen University, Human-Computer Interaction Center
Research Questions & Answers
 Do Decision Support Systems influence operators’ performance?
– Correct Decision Support System reduce Reaction Times
– Correct Decision Support Systems increase Accuracy
 Do operators follow defective Decision Support Systems?
– Limited influence on Reaction Times
– Defective Decision Support Systems still obeyed
and lead to lower accuracy!
 Does the influence of a Decision Support System
relate to the task complexity?
– Effects emerge only for longer tables
 Next steps
– Larger sample (influence of Perceptual Speed, Trust in Automation)
– Validate in Context (Business Simulation Games)
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
page 20

21. RWTH Aachen University, Human-Computer Interaction Center
Thank you four your attention!
Summary
 Industrial Internet leads to increased information complexity
 Supporting the Human-in-the-Loop gains importance
 Business Simulation Games as a Research Tool
 Decision Support Systems crucial for performance
– Operators easily deflected by defective DSS
– Effects only emerge for complex conditions
Dipl.-Inform. Philipp Brauner
Human-Computer Interaction Center
Chair for Communication Science
RWTH Aachen University, Germany
eMail: brauner@comm.rwth-aachen.de
page 21
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016
Funded by the German Research Foundation (DFG) within
the Cluster of Excellence “Integrated Production Technology
for High Wage Countries” (EXC 128).

22. RWTH Aachen University, Human-Computer Interaction Center
 Brauner P, Runge S, Groten M, et al (2013) Human Factors in Supply Chain Management – Decision making in complex logistic scenarios. In: Yamamoto
S (ed) Proceedings of the 15th HCI International 2013, Part III, LNCS 8018. Springer-Verlag Berlin Heidelberg, Las Vegas, Nevada, USA, pp 423–432
 Brauner P (2014) Understanding Human Factors in Supply Chains and Quality Management by Using Business Simulations. In: Brecher C, Wesch-
Potente C (eds) Proceedings of the Conference of the Cluster Of Excellence “Integrative Production Technology For High Wage Countries” 2014/1, 1st
edn. Apprimus Verlag, Aachen, Germany, Aachen, Germany, pp 387–396
 Hering N, Meißner J, Runge S, Brauner P (2014) Exzellenzcluster: Was bestimmt die Performance meiner Supply-Chain? – Eine Untersuchung
technischer und menschlicher Einflussfaktoren im Hinblick auf die Effizienz von Lieferketten. Unternehmen der Zukunft - Zeitschrift für Betriebsorganisation
und Unternehmensentwicklung 27–28.
 Philipsen R, Brauner P, Stiller S, et al (2014a) The role of Human Factors in Production Networks and Quality Management. – How can modern ERP
system support decision makers? First International Conference, HCIB 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27,
2014. Proceedings, LNCS 8527. Springer Berlin Heidelberg, pp 80–91
 Philipsen R, Brauner P, Stiller S, et al (2014b) Understanding and Supporting Decision Makers in Quality Management of Production Networks. Advances
in the Ergonomics in Manufacturing. Managing the Enterprise of the Future 2014 : Proceedings of the 5th International Conference on Applied Human
Factors and Ergonomics, AHFE 2014. CRC Press, Boca Raton, pp 94–105
 Stiller S, Falk B, Philipsen R, et al (2014) A Game-based Approach to Understand Human Factors in Supply Chains and Quality Management. Procedia
CIRP 20:67–73. doi: 10.1016/j.procir.2014.05.033
 Brauner P, Ziefle M (2015) Human Factors in Production Systems – Motives, Methods and Beyond. In: Brecher C (ed) Advances in Production
Technology. Springer International Publishing, pp 187–199
 Mittelstädt V, Brauner P, Blum M, Ziefle M (2015) On the visual design of ERP systems – The role of information complexity, presentation and human
factors. 6th International Conference on Applied Human Factors and Ergonomics and the Affiliated Conferences, AHFE 2015. pp 270–277
 Calero Valdez A, Brauner P, Schaar AK, et al (2015) Reducing Complexity with simplicity - Usability Methods for Industry 4.0. 19thTriennial Congress of
the International Ergonomics Association (IEA 2015).
 Ziefle M, Brauner P, Speicher F (2015) Effects of data presentation and perceptual speed on speed and accuracy in table reading for inventory control.
Occupational Ergonomics 12:119–129. doi: 10.3233/OER-150229
 Brauner, P., Philipsen, R., Fels, A., Fuhrmann, M., Ngo, H., Stiller, S., Schmitt, R., Ziefle, M.: A Game-Based Approach to Meet the Challenges of Decision
Processes in Ramp-Up Management. Quality Management Journal. 23, 55–69 (2016).
 Calero Valdez, A., Brauner, P., Ziefle, M.: Preparing Production Systems for the Internet of Things The Potential of Socio-Technical Approaches in Dealing
with Complexity. In: Dimitrov, D. and Oosthuizen, T. (eds.) Proceedings of the 6th International Conference on Competitive Manufacturing 2016 (COMA
’16). pp. 483–487. CIRP, Stellenbosch, South Africa (2016).
 Brauner P, Ziefle M. How to train employees, identify task-relevant human factors, and improve software systems with Business Simulation Games.
Procedings of the International Conference on Competitive Manufacturing 2016, COMA ’16. Stellenbosch, SA; 2016. p. 541–6.
 Brauner, P., Calero Valdez, A., Philipsen, R., Ziefle, M.: Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s
Performance in Production Environments. Proceedings of the Human-Computer Interaction International 2016. (in press)
 Brauner, P., Philipsen, R., Ziefle, M.: Projecting Efficacy and Use of Business Simulation Games in the Production Domain using Technology Acceptance
Models. Proceedings of the Applied Human Factors and Ergonomics Conference (AHFE 2016). (in press)
 Brauner, P., Philipsen, R., Calero Valdez, A., Ziefle, M.: On Studying Human Factors in Complex Cyber-Physical Systems, Workshop HFIDSS 2016,
Mensch &Computer 2016 (in press)
Publications
page 22
2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems
Influences User’s Performance in Production Environments" - HCII 2016