The document discusses a proposed system called Smart Cabs that uses sensors and pattern recognition to monitor a machine operator's state of mind. The system would analyze factors like tiredness, alertness, and happiness. It would then make minor adjustments to the operating environment or provide encouragement to improve productivity and worker well-being. A demonstration would use sensors to estimate an operator's emotions in real-time and simulate corrective actions. The data could also be used long-term to optimize job design and practices. The approach focuses on quick iterations to develop a cross-platform prototype that connects operator state of mind data to machine performance data.

1. SMART CABS:
MACHINES THAT
KNOW THEIR
DRIVERS
Rod Walsh, Petri Murtomaki, and
Kimmo Vänni
TAMK
version
0.1
date
15.03.2012
author
RW & KV
details
Created & first ideas
Demola InnoSummer 2012
0.2 16.03.2012 Rod Walsh Minor improvements
0.3 02.05.2012 Rod Walsh Filled out the “complete story”
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 1

2. COMING UP IN THIS SLIDE SET…
 Why: Better Performance in Forestry
 What: The Human Touch
 How: The Demo
 Where: The Big Idea
 Approach: Approach
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 2

3. BETTER PERFORMANCE IN FORESTRY
 The commercial performance of large human-operated machines is
largely determined by the performance of the human operator
 Today, human operator performance is largely driven by hard external
factors, such as training, experience and attitude
 Dynamic factors are “left to care for themselves”: such as tiredness,
alertness, attentiveness, happiness, etc.
 But we want to use technology and human-insight to monitor these soft
internal factors
 And improve working life, long-term health and commercial productivity
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 3

4. THE HUMAN TOUCH non-contact sensing
 We will take a look at the emotional state-of-mind of
operators using face, sound and posture monitoring
technology with pattern recognition Psychology
& processing
 And use our knowledge of these soft internal factors for
improvements: state of mind
 Happier and lower-stress work (short and long term benefit for the
employee)
 Better productivity (short and long term benefit for the employer)
 By:
 Dynamically modifying the working environment for the better
(short term)
 Identifying positive patterns of emotion affect on human Simple changes
performance & motivation, and then matching practices, assignments • Music, lighting, airflow, …
and environments the patterns (long-term)
Pattern Working
recognition practices
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 4

5. Examples of “state of mind”
THE DEMO • Tiredness
• Boredom
• Willingness to work
• Fear/anxiety
 Multiple HD webcams, microphones and PrimeSense IR sensors (e.g. • Happiness
Kinect) will be arranged to monitor a human “operator” (non-contact
sensing) Examples of corrective action:
 (For versatility, an “office desk operator” setup is needed. The team may take • Encouragement
physical forestry machine mock-ups and closeness-to-reality to higher levels.) • Stimulation
 A set of “states of mind” that are relevant to machine operator • Pause/end of task
performance and wellbeing will be selected • Verify the measurement
 Quickly selected emotions at first (for rapid development) & then iterated Examples of Job improvements:
 Sensor signals are classified for the “states of mind” • Productivity
 Classifier(s) will be “trained” and tested. Training and testing will begin with
• Volume
“acted emotions” and tightly iterated between the pattern recognition and • Errors
the pyschology/emotional model. • Motivation for the job
 Offline: all sensor and analytics data will be logged, to allow discovery • Intervention before
of longer-term patterns (such as time of day patterns) problems become critical
 Real-time: The instantiations state-of-mind is matched against a “task
model” and need for corrective action (on the operator) is
calculated
 As determined, corrective action is taken to change the operator’s
environment
 The effects and affects are logged to determine whether the action succeeded
 (The “office desk simulator” can be a PC display simulation, or better…)
SEE NEXT SLIDE FOR VISUAL DESCRIPTION
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 5

6. THE DEMO Database: non-contact sensing:
state of mind log video, image, audio
Pattern sensor logs
recognition
logged
logged offline
real-time
7/10
capability
state of mind
estimation
~7/10
capability
Match
with
task Simulate simple changes
8/10 • Music, lighting, airflow, …
minimum
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 6

7. THE BIG PICTURE
 For the long-term benefits, the data can be used to change the design of
working environments and practices, so…
 The demo would be integrated to a larger system (see next slide)
 Existing telematics data from the forestry machines can introduced to
the common database and analyzed for patterns between operator
state of mind and machine behavior (for further insights and causalities)
 This is beyond what the team needs to do!
 The team’s innovation and excitement decide what is done beyond the core demo
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 7

8. Human impact
Database:
THE BIG PICTURE
on work
quality & telematics log
productivity & state of mind log
machine sensor logs
performance
state of mind
Machine + Logging telematics
environment (exists already)
impact on capability
human
operator
capability
Match
Improvements
required
corrective action
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 8

9. APPROACH
 In theory, the team is free to adopt any approach that:
 Works well, looks great and receives “ooh” and “wow” sounds
 Fits the objectives
 Is reusable, extendable and portable (as a whole and as components)
 Meeting these needs in one go is near impossible, so iteration, communication
and sharing are critical – and at high speed!
 In practice, the support team has some useful experience and advice:
 Short design, implementation and demo iterations are the safest and coolest
 Stick to technologies which are cross-platform and open (when possible):
 E.g. HTML5, OpenNI, Published solutions, etc. as applicable
 We will supply USB webcams (inc. microphones) and PrimeSense IR sensors
 Code should be runnable on Mac/Win/Linux (Ubuntu is our favorite Linux)
 We will workshop together to best use the team’s and the support team’s knowledge
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 9

10. Some support slides
…

11. Together with another awesome project,
SMART CAB + we could close the loop on emotional
feedback (possible project extension)
AFFECTIVE ROBOTS
state of mind
state of mind
Goal estimation
Match
Emotive
“corrective” commands
action using and like:
emotionally-savvy • be happy
avatar • welcome
1. Perform the emotion
• Reject
• cry 2. Perform for the emotion
What setting or stage would unlock,
actual robot virtual robot 3. Read/write emotion?
emphasize or inhibit which affects?
body
face
Full
modeled human-like emotion
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 11

12. Human impact non-contact sensing:
on work Database:
telematics log video, image, audio
quality &
productivity & Pattern state of mind log
machine recognition sensor logs
logged
performance
logged offline
“state of mind”
Machine + Logging telematics real-time
environment (exists already)
impact on
7/10
capability
human
operator state of mind
estimation
~7/10
capability
Match
with
task Simulate simple changes Job improvements
8/10 • Music, lighting, airflow, …
minimum
corrective action
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 12

13. SIMPLE ONE-SLIDER
Design of
practices &
environment
telematics
© TAMK, 2012. ALL RIGHTS RESERVED. TAMK CONFIDENTIAL. 13