The document describes a standardized method for using micro tasks on a tablet to efficiently and objectively benchmark innovative control room displays against conventional interfaces. Operators complete time-sensitive questions linked to simulator scenarios under different display conditions to generate quantitative performance data. A 2015 study found innovative displays led to faster response times and slightly better accuracy than conventional displays. The method provides a highly customizable, precise way to generate data needed for human reliability assessments and can be used for training, evaluation, and benchmarking across organizations.

1. micro tasks
s w i p e t o s t a r t

2. our goal: a standardized method for
benchmarking control room interfaces
• Tablet tool for data collection
• Can be used stand-alone or
• Linked to simulator
• Standardized data collection procedure
• Standardized method for question generation
• Standardized set of questions
• Database

3. micro tasks are for
 Evaluation
 HRA
 Training

4. Efficiently and objectively benchmark innovative
displays against conventional interfaces

5. IFE process overview
display on tablet
Example
IFE will conduct a micro task evaluation of this set-up
in December 2015 at a U.S. training simulator

6. Compare

7. IFE design concept for overview displays on tablet,
developed for the 2015 U.S. simulator study (screen 1 of 3)

8. example
innovative vs. conventional

9. mass balance
conventional innovative
Do the innovative displays lead to faster,
more reliable identifications and decisions?

10. mass balance
conventional innovative
Is performance (time and reliability) with innovative
displays at least as good as with conventional?

11. How to test performance benefits
of new interface solutions?

12. Scenario-based methods.
Observational or self-report.
Qualitative insights.
Relatively few data points.
Decontextualised.
Performance based.
Quantitative data.
Large amounts of data.
flavors of t&e methods

13. micro tasks are…

14.  Large number of questions
Related to systems, components, procedures, etc.
 Varying levels of difficulty
Including higher-level decision making
 Different display conditions
E.g. innovative vs. conventional displays
 We measure response time / accuracy
 Compare data between conditions
If needed, review eye tracking recordings
to understand anomalies
detection/decision tasks under time pressure

15.  Micro task tablet app linked to simulator
Tablet app receives signals from simulator
Tablet app can send signals / commands to simulator
 Dynamic scenarios
To cover monitoring / vigilance tasks
 We can now record operator actions
Can record operating of components, e.g. “start RCPs”
 New system for aggregating data
 New system for managing task lists
Makes it easier to set up and manage a study
new in 2015

16. example

18. 4 operators
working in the
simulator
individually
(no communication)

19. We control
which displays
are available
(eg innovative, conventional)

20. Instructions:
”Please answer the
questions correctly,
but also as quickly
as possible.
It is very important
that you work as
fast as you can.”

21. Run 1
s w i p e t o s t a r t

22. How many condensate pumps
are running on turbine 31?
1 2 3
s w i p e t o
c o n t i n u e

23. “How many condensate pumps
are running on turbine 31?”
Average identification time
Conventional 8 sec
Innovative 6 sec

24. Is the subcooling margin sufficient?
Yes
s w i p e t o
c o n t i n u e
No

25. Which steam generators
are faulty?
SG-1
s w i p e t o
c o n t i n u e
SG-2 SG-3

26. Should safety injection
be stopped?
Yes
s w i p e t o
c o n t i n u e
No

27. What is the narrow-range level
in steam generator 1?
s w i p e t o
c o n t i n u e
%
1 2 3 4 5 6 7 8 9 0
37

28. up to 200 questions per hour
with 4 operators in the simulator working individually,
that means we can run up to 800 questions per hour

29. Tablet can trigger events in the simulator (e.g. start a tube leak)
and receive signal from the simulator (e.g. RCP-2 was started)

30. video
https://vimeo.com/131387407

31. results

32. data source
where does this data come from?

33. 6840 data points
3420 response time measures
3420 accuracy measures

34. 20 operators

35. 5 hours simulator time
in total for the whole data collection

36. innovative displays are superior

37. HSI mean performance times
Current effect: F(2, 8,003 )=21,889, p=,00057
Vertical bars denote 0,95 confidence intervals
LSD (Innovative) OWD (Conventional) LSD and OWD
15
16
17
18
19
20
21
22
23
24
25
26
Performancetime(sec)
Operators were faster with innovative
displays than with conventional displays
innovative
conventional
HSI mean performance times
Current effect: F(2, 8,003 )=21,889, p=,00057
Vertical bars denote 0,95 confidence intervals
21
22
23
24
25
26
etime(sec)
Highly statistically
significant difference

38. Slightly more accurate with innovative
displays than conventional displays
Percentage of correct answers
innovative conventional innovative

39. Performance variability

40. conclusion

41.  As reliable as conventional displays
 Faster to read
innovative displays in the 2014 study are

42. summary

43.  Generate quantitative results about
performance benefits (time and reliability)
of digital/innovative CR interfaces
 Highly efficient, objective method
 Compare to existing reference data
 Qualitative insights via eye tracking
 Data directly supports HRA
 Highly customisable and precise
Generate exactly the data needed for the HRA

44. big picture
micro task tablet app = a mobile companion for
Human Factors / Human Reliability specialists

45. Micro task
tablet app
Standardized
question sets
Database
training
hra interface
design
gamification
evaluation of training
validation
benchmarking
within / across organisations

46. what next?

47. Benchmarking studies 2015 / 2016

48. benchmarking studies
(1) 2015 study at U.S. training simulator
analog interfaces vs IFE displays
(2) 2016 study of a partially digital control room
analog interfaces vs. 1990s-vintage overview
display vs IFE display
(3) 2016 study of a fully digital control room
Measure operator performance and reliability in a
fully digital control room and benchmark against
data from study 1 and 2

49.  Micro task database
 Micro tasks for training
Includes gamification approach
 Micro tasks for actions outside the
control room (field operator, mechanical, etc)
roadmap

50.  Micro tasks for team decisions
 Add secondary task capability
 Add workload measurement capability
roadmap

51. michael.hildebrandt@ife.no