Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments

65 views

Published on

Despite the increasing automation levels in an Industry 4.0 scenario, the tacit knowledge of highly skilled manufacturing workers remains of strategic importance. Retaining this knowledge by formally capturing it is a challenge for industrial organisations. This paper explores research on automatically capturing this knowledge by using methods from activity recognition and process mining on data obtained from sensorised workers and environments. Activity recognition lifts the abstraction level of sensor data to recognizable activities and process mining methods discover models of process executions.
We classify the existing work, which largely neglects the possibility of applying process mining, and derive a taxonomy that identifies challenges and research gaps.

Published in: Science
  • Be the first to comment

  • Be the first to like this

A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments

  1. 1. A TAXONOMY FOR COMBINING ACTIVITY RECOGNITION AND PROCESS DISCOVERY IN INDUSTRIAL ENVIRONMENTS Felix Mannhardt, Riccardo Bovo, Manuel Fradinho Oliveira, Simon Julier 1SINTEF, Trondheim, Norway , 2University College London, UK This research has received funding from the European Unions H2020 research and innovation programme under grant agreement no. 723737 (HUMAN).
  2. 2. What is Process Mining? 2 • Activity (What?) • Ordering (When?) • Case / Context • SAP, MES, … • Sensors (IoT) • … • Assembly • Logistics • … • Work instructions • Assembly sequence • …
  3. 3. 3 Operators Sensor data discover Process Models capture Activities recognise analyse tacit knowledge deliver explicit knowledge Process Engineer Process Discovery in Industrial Environments
  4. 4. What is Activity Recognition (ARC)? 4 Some input sensor signal segmented according to the ground truth label
  5. 5. 5 Example: Supervised detection using a LSTM
  6. 6. 6 Some further typical examples Ekaterina H. Spriggs*, Fernando De La Torre, and Martial Hebert, "Temporal segmentation and activity classification from first-person sensing," IEEE Workshop on Egocentric Vision, CVPR 2009, June, 2009. Michael S. Ryoo Greg Mori Kris Kitani. CVPR1 Tutorial: Paper 18: Human Activity Recognition. 2018
  7. 7. Project Context – HUMAN • Human MANufacturing Project (H2020) • Sensorised manufacturing operator (Wearables etc.) • Stress and fatigue detection through sensor data • Support operators in their workflow through technology • Augmented reality (AR) • Exoskeletons • Use of Process Mining on collected data for long-term diagnostics and optimisation of work 7 Challenge: Automatically detect manual shop-floor activities to re-discover the actual work process
  8. 8. Exploratory search for existing work 8 Google Scholar Scopus Search terms • Manufacturing • Industrial • Activity recognition • Sensors • Process discovery • Process elicitation • Process analysis • Events • Tacit knowledge
  9. 9. Categorized 26 relevant papers 9
  10. 10. Preliminary taxonomy for ARC & Process discovery 10 Time Predictive Online Post-mortem
  11. 11. 11 Knoch, S., Ponpathirkoottam, S., Fettke, P., Loos, P.: Technology-enhanced process elicitation of sworker activities in manufacturing. In: Business Process Management Workshops, pp. 273–284. Springer International Publishing (2018) Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tr, G.: Wearable activity tracking in car manufacturing. IEEE Pervasive Computing 7(2), 42–50 (2008) Ambient vs. Wearable | Video vs. Motion
  12. 12. Preliminary taxonomy for ARC & Process discovery 12 Data Capture Sensor type Vision-based Motion-based Sound-based Radiowave-based Sensor location Wearable Objects Ambient Storage Batch Streaming Processing Supervised Unsupervised
  13. 13. Very little unsupervised learning 13 Boettcher, S., Scholl, P.M., Laerhoven, K.V.: Detecting process transitions from wearable sensors. In: iWOAR 2017. ACM Press (2017)
  14. 14. Some semi-supervised works 14 Blanke, U., Schiele, B.: Remember and transfer what you have learned - recognizing composite activities based on activity spotting. In: ISWC 2010. IEEE (2010)
  15. 15. Preliminary taxonomy for ARC & Process discovery 15 Processcontext Activity type Granularity Coarse Fine Complexity Simple Complex Control-flow type Sequence of steps Sequential workflow Collaborative workflow Process size Small Large
  16. 16. 16 Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tr, G.: Wearable activity tracking in car manufacturing. IEEE Pervasive Computing 7(2), 42–50 (2008) Complexity & Control-Flow Blanke, U., Schiele, B.: Remember and transfer what you have learned - recognizing composite activities based on activity spotting. In: ISWC 2010. IEEE (2010)
  17. 17. Preliminary taxonomy for ARC & Process discovery 17 Environment Setting Laboratory Factory Layout Flexible Structured RigidInterference Privacy
  18. 18. 18 Voulodimos, A.S., Kosmopoulos, D.I., Doulamis, N.D., Varvarigou, T.A.: A top-down event driven approach for concurrent activity recognition. Multimedia Tools and Applications 69(2), 293–311 (2012) Environment Knoch, S., Ponpathirkoottam, S., Fettke, P., Loos, P.: Technology-enhanced process elicitation of sworker activities in manufacturing. In: Business Process Management Workshops, pp. 273–284. Springer International Publishing (2018)
  19. 19. Conclusion • Contribution • A structured overview of the work on activity recognition relevant to process discovery in industrial environments • Preliminary taxonomy derived from our literature search • Challenges identified for process discovery in industrial environments • Future work • Rigerous description of the finalised taxonomy • Comprehensive literature review based on the completed taxonomy • Validation of the taxonomy? • Apply process discovery paired with activity recognition in real industrial environments 19
  20. 20. Mannhardt F., Bovo R., Oliveira M.F., Julier S. (2018) A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments. In: Yin H., Camacho D., Novais P., Tallón-Ballesteros A. (eds) Intelligent Data Engineering and Automated Learning – IDEAL 2018. IDEAL 2018. Lecture Notes in Computer Science, vol 11315. Springer, Cham

×