A TAXONOMY FOR COMBINING
ACTIVITY RECOGNITION
AND
PROCESS DISCOVERY
IN INDUSTRIAL ENVIRONMENTS
Felix Mannhardt, Riccardo Bovo, Manuel Fradinho Oliveira, Simon Julier
1SINTEF, Trondheim, Norway , 2University College London, UK
This research has received funding from the European
Unions H2020 research and innovation programme under grant
agreement no. 723737 (HUMAN).

What is Process Mining?
2
• Activity (What?)
• Ordering (When?)
• Case / Context
• SAP, MES, …
• Sensors (IoT)
• …
• Assembly
• Logistics
• …
• Work instructions
• Assembly sequence
• …

3
Operators
Sensor
data
discover
Process
Models
capture
Activities
recognise
analyse
tacit knowledge
deliver explicit knowledge
Process
Engineer
Process Discovery in Industrial Environments

What is Activity Recognition (ARC)?
4 Some input sensor signal segmented according to the ground truth label

5
Example: Supervised detection using a LSTM

6
Some further typical examples
Ekaterina H. Spriggs*, Fernando De La Torre, and Martial Hebert, "Temporal segmentation and activity
classification from first-person sensing," IEEE Workshop on Egocentric Vision, CVPR 2009, June, 2009.
Michael S. Ryoo Greg Mori Kris Kitani.
CVPR1 Tutorial: Paper 18: Human Activity Recognition. 2018

Project Context – HUMAN
• Human MANufacturing Project (H2020)
• Sensorised manufacturing operator (Wearables etc.)
• Stress and fatigue detection through sensor data
• Support operators in their workflow through technology
• Augmented reality (AR)
• Exoskeletons
• Use of Process Mining on collected data for
long-term diagnostics and optimisation of work
7
Challenge: Automatically detect manual shop-floor activities
to re-discover the actual work process

Exploratory search for existing work
8
Google Scholar Scopus
Search terms
• Manufacturing
• Industrial
• Activity recognition
• Sensors
• Process discovery
• Process elicitation
• Process analysis
• Events
• Tacit knowledge

Categorized 26 relevant papers
9

Preliminary taxonomy for ARC & Process discovery
10
Time
Predictive
Online
Post-mortem

11
Knoch, S., Ponpathirkoottam, S., Fettke, P., Loos, P.:
Technology-enhanced process elicitation of sworker activities in manufacturing.
In: Business Process Management Workshops,
pp. 273–284. Springer International Publishing (2018)
Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tr, G.:
Wearable activity tracking in car manufacturing.
IEEE Pervasive Computing 7(2), 42–50 (2008)
Ambient vs. Wearable | Video vs. Motion

Preliminary taxonomy for ARC & Process discovery
12
Data
Capture
Sensor type
Vision-based
Motion-based
Sound-based
Radiowave-based
Sensor location
Wearable
Objects
Ambient
Storage
Batch
Streaming
Processing
Supervised
Unsupervised

Very little unsupervised learning
13
Boettcher, S., Scholl, P.M., Laerhoven, K.V.:
Detecting process transitions from wearable sensors.
In: iWOAR 2017. ACM Press (2017)

Some semi-supervised works
14
Blanke, U., Schiele, B.:
Remember and transfer what you have learned - recognizing composite
activities based on activity spotting.
In: ISWC 2010. IEEE (2010)

Preliminary taxonomy for ARC & Process discovery
15
Processcontext
Activity type
Granularity
Coarse
Fine
Complexity
Simple
Complex
Control-flow type
Sequence of steps
Sequential
workflow
Collaborative
workflow
Process size
Small
Large

16
Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tr, G.:
Wearable activity tracking in car manufacturing.
IEEE Pervasive Computing 7(2), 42–50 (2008)
Complexity & Control-Flow
Blanke, U., Schiele, B.:
Remember and transfer what you have learned - recognizing composite
activities based on activity spotting.
In: ISWC 2010. IEEE (2010)

Preliminary taxonomy for ARC & Process discovery
17
Environment
Setting
Laboratory
Factory
Layout
Flexible
Structured
RigidInterference
Privacy

18
Voulodimos, A.S., Kosmopoulos, D.I., Doulamis, N.D., Varvarigou, T.A.:
A top-down event driven approach for concurrent activity recognition.
Multimedia Tools and Applications
69(2), 293–311 (2012)
Environment
Knoch, S., Ponpathirkoottam, S., Fettke, P., Loos, P.:
Technology-enhanced process elicitation of sworker activities in manufacturing.
In: Business Process Management Workshops,
pp. 273–284. Springer International Publishing (2018)

Conclusion
• Contribution
• A structured overview of the work on activity recognition
relevant to process discovery in industrial environments
• Preliminary taxonomy derived from our literature search
• Challenges identified for process discovery in industrial environments
• Future work
• Rigerous description of the finalised taxonomy
• Comprehensive literature review based on the completed taxonomy
• Validation of the taxonomy?
• Apply process discovery paired with activity recognition in real industrial environments
19

Mannhardt F., Bovo R., Oliveira M.F., Julier S. (2018)
A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments.
In: Yin H., Camacho D., Novais P., Tallón-Ballesteros A. (eds)
Intelligent Data Engineering and Automated Learning – IDEAL 2018. IDEAL 2018.
Lecture Notes in Computer Science, vol 11315. Springer, Cham