Triggering Proactive Business Process Adaptations via Online Reinforcement Learning
•Download as PPTX, PDF•
0 likes•71 views
Proactive process adaptation can prevent and mitigate upcoming problems during process execution by using predictions about how an ongoing case will unfold. There is an important trade-off with respect to these predictions: Earlier predictions leave more time for adaptations than later predictions, but earlier predictions typically exhibit a lower accuracy than later predictions, because not much information about the ongoing case is available. An emerging solution to address this trade-off is to continuously generate predictions and only trigger proactive adaptations when prediction reliability is greater than a predefined threshold. However, a good threshold is not known a priori. One solution is to empirically determine the threshold using a subset of the training data. While an empirical threshold may be optimal for the training data used and the given cost structure, such a threshold may not be optimal over time due to non-stationarity of process environments, data, and cost structures. Here, we use online reinforcement learning as an alternative solution to learn when to trigger proactive process adaptations based on the predictions and their reliability at run time. Experimental results for three public data sets indicate that our approach may on average lead to 12.2% lower process execution costs compared to empirical thresholding.
Recommended
More Related Content
Similar to Triggering Proactive Business Process Adaptations via Online Reinforcement Learning
Similar to Triggering Proactive Business Process Adaptations via Online Reinforcement Learning (20)
More from Andreas Metzger
More from Andreas Metzger (13)
Recently uploaded
Recently uploaded (20)
Triggering Proactive Business Process Adaptations via Online Reinforcement Learning
- 1. Triggering Proactive Business Process Adaptations via Online Reinforcement Learning Andreas Metzger, Tristan Kley, Alexander Palm 18th Int. Conference on Business Process Management (BPM 2020) A. Metzger, T. Kley, and A. Palm, “Triggering proactive business process adaptations via online reinforcement learning,” in 18th Int’l Conference on Business Process Management (BPM 2020), Sevilla, Spain (virtual), September 13-18, 2020, ser. LNCS, D. Fahland, C. Ghidini, J. Becker, and M. Dumas, Eds., vol. 12168. Springer, 2020. https://doi.org/10.1007/978-3-030-58666-9_16
- 3. Process completionPrediction point j Process start Proactive Process Adaptation “in a Nutshell” BPM 20203 Monitor Predict Proactive adaptation planned / acceptable situations = Violation = Non- Violation Process Performance
- 4. Requirements for Proactive Process Adaptations Requirement 1: Accuracy of Predictions • False violations Unnecessary adaptations • False non-violations Missed adaptations Requirement 2: Earliness of Predictions • Early predictions More time for adaptations But: trade-off between accuracy and earliness BPM 20204 BPIC 2012 [Teinemaa et al. 2019]Cargo 2000 [Metzger et al., 2020a] Accuracy Prediction point C1 C3 C5 C7 C9 Low High Early Late BPIC 2017 [Wang et al. 2019]
- 5. Trading off Accuracy and Earliness (State of the Art) Reliability-based Adaptation • Use first prediction with sufficient reliability • Class probability of random forests [Teinemaa et al. 2018] • Majority count of deep neural network ensembles [Metzger et al. 2019] • Sufficient reliability expressed via threshold • But: optimal threshold is unknown! Empirical Thresholding [Teinemaa et al. 2018; Fahrenkrog-Petersen et al. 2019] • Determine “optimal” threshold using dedicated training data set • But: Threshold may not remain optimal due to non-stationarity in process environments, data or cost structures BPM 20205 Threshold Time of Adaptation Risk for Adaptation Lower Earlier More wrong adaptations Higher Later Less time for adaptations
- 7. Overall Approach (Simplified) Learn at run-time when to trigger adaptations BPM 20207 Monitoring data at prediction point j Reliability Estimate j Relative, Predicted Deviation j Process Execution (BPM System) Online Reinforcement Learning (RL) Predictive Process Monitoring (Ensemble Deep Supervised Learning) Action (triggering proactive adaptation) Reward Relative Prefix Length j State (Obser- vation)
- 8. Overall Approach BPM 20208 Monitoring data at prediction point j RNN-LSTM Model 1 RNN-LSTM Model m … Reliability Estimate j Relative, Predicted Deviation jyj,1 ^ yj,m ^ a r Action Selection (Sampling) Action Selection Policy (Artificial Neural Network) s Policy Update (Gradient Ascent) s’ a Process Execution (BPM System) Action (triggering proactive adaptation) Reward RNN-LSTM Model 2 yj,2 ^ Relative Prefix Length j
- 9. Online Reinforcement Learning Technology Foundation: Policy-based RL [Palm et al., 2020] • No need to quantize states (like in tabular Q-Learning) • No need to fine-tune exploration vs. exploitation (like in Q-Learning / SARSA) -- especially no need to re-tune due to non-stationarity Formulating the Learning Problem • Strong rewards r (= learning goals) • Rewards modelled closely to actual problem domain may be weak; e.g., successful process execution zero costs r = 0 • successful process execution r = +1 • Episode = execution of a single case • Actual rewards r at end of each case BPM 20209 1 2 3 4 5 No adaptation r2 = 0 No adaptation r3 = 0 Adaptation r4 = r
- 11. -0,4 -0,3 -0,2 -0,1 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 100 355 610 865 1120 1375 1630 1885 2140 2395 2650 2905 3160 3415 3670 3925 4180 4435 4690 4945 5200 5455 5710 5965 6220 6475 6730 6985 7240 7495 7750 8005 8260 8515 8770 9025 9280 9535 9790 10045 10300 Learning behaviour BPM 202011 Rate of adaptations Earliness Reward/100 Rate of correct adaptations BPIC 2017 Episodes (= cases)
- 12. Learning behaviour BPM 202012 -0,2 -0,1 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 100 1101 2102 3103 4104 5105 6106 7107 8108 9109 10110 11111 12112 13113 14114 15115 16116 17117 18118 19119 20120 21121 22122 23123 24124 25125 26126 27127 28128 29129 30130 31131 32132 33133 34134 35135 36136 37137 38138 39139 40140 41141 42142 43143 44144 45145 46146 47147 48148 49149 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 98 1099 2100 3101 4102 5103 6104 7105 8106 9107 10108 11109 12110 13111 14112 15113 16114 17115 18116 19117 20118 21119 22120 23121 24122 25123 26124 27125 28126 29127 30128 31129 32130 33131 34132 35133 36134 37135 38136 39137 40138 41139 42140 43141 44142 45143 46144 47145 48146 49147 Accuracy (avg MAE; last 100 cases) Accuracy (avg MAE; last 100 cases) Accuracy (MAE) Traffic
- 13. Quantitative Results Cost Model [Teinemaa et al., 2018; Metzger et al., 2019] • Penalty • Adaptation / compensation costs = * Penalty • Adaptation effectiveness Average cost savings compared with empirical thresholding Improvement of using strong reward function • 36.9 % lower costs for BPIC 2017 [Palm et al. 2020] BPM 202013 Data Set min = .5 min = 0 BPIC 2012 2.0% 3% BPIC 2017 1.0% 11.4% Traffic 24.3% 32.1% Average 9.1% 15.5% (Erratum: Average cost savings reported in paper are smaller due to missing value for = .4)
- 15. Conclusion and Outlook Online Reinforcement Learning for effective decision support for proactive process adaptation Future enhancements • Consider multiple adaptation options [Fahrenkrog-Petersen et al., 2019] • Increase speed of convergence • Perform offline pre-training during design time • Better guide exploration in presence of multiple adaptation options [Metzger et al., 2020b] • Explain predictions and adaptation decisions [Sindhgatta et al. 2020, Mehdiyev and Fettke, 2020] 15 BPM 2020 Research leading to these results has received funding from the EU’s H2020 research and innovation programme under grant agreements no. 871493 – https://dataports-project.eu/ 780351 – https://enact-project.eu/ Thank You! -0,4 -0,3 -0,2 -0,1 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 100 355 610 865 1120 1375 1630 1885 2140 2395 2650 2905 3160 3415 3670 3925 4180 4435 4690 4945 5200 5455 5710 5965 6220 6475 6730 6985 7240 7495 7750 8005 8260 8515 8770 9025
- 16. References [Metzger et al., 2019] A. Metzger, A. Neubauer, P. Bohn, and K. Pohl, “Proactive process adaptation using deep learning ensembles,” in CAiSE 2019, LNCS, , vol. 11483. Springer, 2019. [Teinemaa et al. 2019] Teinemaa, I., Dumas, M., Rosa, M.L., Maggi, F.M.: Outcome-oriented predictive process monitoring: Review and benchmark. ACM Trans. Knowledge Discovery from Data (TKDD) 13, 2019 [Metzger et al., 2020a] A. Metzger, J. Franke, and T. Jansen, “Ensemble deep learning for proactive terminal process management at the port of Duisburg ’duisport’,” in Business Process Management Cases – Volume 2, J. vom Brocke, J. Mendling, and M. Rosemann, Eds. Springer, 2020 – to be published. [Metzger et al., 2020b] A. Metzger, C. Quinton, Z. Mann, L. Baresi, and K. Pohl, “Feature model-guided online reinforcement learning for self-adaptive services,” in 18th Int’l Conference on Service-Oriented Computing (ICSOC 2020), LNCS, Springer, 2020 – to be published. [Wang et al., 2019] J. Wang, D. Yu, C. Liu, and X. Sun, “Outcome-oriented predictive process monitoring with attention- based bidirectional LSTM neural networks,” in ICWS 2019, IEEE, 2019 [Teinemaa et al., 2018] Teinemaa, I., Tax, N., de Leoni, M., Dumas, M., Maggi, F.M.: Alarm-based prescriptive process monitoring. In: Weske, M., Montali, M., Weber, I., vom Brocke, J. (eds.) BPM Forum 2018, LNBIP, vol. 329, Springer, 2018 [Fahrenkrog-Petersen et al., 2019] Fahrenkrog-Petersen, S.A., Tax, N., Teinemaa, I., Dumas, M., de Leoni, M., Maggi, F.M., Weidlich, M.: Fire now, fire later: Alarm-based systems for prescriptive process monitoring. CoRR abs/1905.09568, 2019 [Francescomarino et al., 2017] Francescomarino, C.D., Ghidini, C., Maggi, F.M., Petrucci, G., Yeshchenko, A.: “An eye into the future: Leveraging a-priori knowledge in predictive business process monitoring“, in BPM 2017, LNCS, vol. 10445, Springer, 2017 [Palm et al., 2020] A. Palm, A. Metzger, and K. Pohl, “Online reinforcement learning for self-adaptive information systems,” in CAiSE 2020, LNCS, vol. 12127, Springer, 2020 [Mehdiyev and Fettke, 2020] N. Mehdiyev and P. Fettke, “Prescriptive process analytics with deep learning and explainable artificial intelligence,” in ECIS 2020 [Sindhgatta et al. 2020] R. Sindhgatta, C. Moreira, C. Ouyang, A. Barros: Exploring Interpretable Predictive Models for Business Processes. BPM 2020: 257-272 BPM 202016
Editor's Notes
- MCC metric for Cargo 2000 (Metzger and Neubauer 2018) AUC metric for BPIC2012 (Teinemaa et al. 2019) BPIC 2017 (Wang et al. 2019). The different curves represent different concrete configurations of the prediction models (e.g., feature encoding)
- RNN-LSTM, increasingly used; e.g., Tax or Evermann
- Note: small mistake in the paper, that the averages where not recomputed in the underlying XLS sheet