Your SlideShare is downloading. ×
0
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

S-CUBE LP: Runtime Prediction of SLA Violations Based on Service Event Logs

349

Published on

Published in: Education, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
349
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. S-Cube Learning Package Service Level Agreements:Runtime Prediction of SLA Violations Based on Service Event Logs TU Wien (TUW), University of Stuttgart (USTUTT) Philipp Leitner, TUW www.s-cube-network.eu
  • 2. Learning Package Categorization S-Cube SBA Quality Management Quality Assurance and Quality Prediction Runtime Prediction of SLA Violations Based on Service Event Logs © S-Cube
  • 3. Learning Package Overview Problem Description Event-Based Runtime Prediction Discussion Conclusions © S-Cube
  • 4. Let’s Consider a Scenario (1) Assume you are a provider of a composite service Commercial customers order products via a Web service interface Your process checks an internal stock, orders some missing parts, and assembles the product [everything Get Payment Charge available] Prefs Customer Receive Select Order Supplier Check Order From Stock [no supplier Ship Order Supplier 1 available] Order From Supplier 2 Cancel Order © S-Cube
  • 5. Let’s Consider a Scenario (2) (Of course) there are contractual obligations – Delivery in time – Availability of your service – Product quality These can be formulated using Service Level Agreements – Note: in this context not necessarily WSLA style machine processable SLAs – can also be just regular contracts [everything available] Get Payment Charge Prefs Customer Receive Select Order Supplier Check Order From Stock [no supplier Ship Order Supplier 1 available] Order From Supplier 2 Cancel Order © S-Cube
  • 6. Let’s Consider a Scenario (3) As a provider, you want to receive timely notifications if these obligations are likely to be violated – Usually there are penalties to be paid if contractual obligations are (repeatedly) violated – Even if this is not the case, customer satisfaction will suffer and existing customers may terminate their contracts Based on this notifications … ! [everything – … countermeasures can be taken (i.e., Receive Select available] Get Payment Charge Prefs Customer adaptations can be triggered) Order Supplier – … customers can be notified Check Order From Stock [no supplier Ship Order Supplier 1 available] Order From Supplier 2 Cancel Order © S-Cube
  • 7. Learning Package Overview Problem Description Event-Based Runtime Prediction Discussion Conclusions © S-Cube
  • 8. One S-Cube Approach:Predictions from Event-Log Data1. Define Checkpoints in the service composition2. If checkpoints are passed, collect all (important and available) runtime data …3. … enrich with estimations for missing data (if possible) …4. … and use Machine Learning techniques to generate predictions for objectives © S-Cube
  • 9. Necessary Inputs Evidently, there are some (not unreasonable) assumptions: 1. The provider needs to define a sensible checkpoint for prediction (  quality / timeliness tradeoff ! ) 2. Important runtime data needs to be defined and monitored 3. Optimally, some estimations for data still missing in the checkpoint is available 4. The system needs to be initialized with some historical data (  to learn the Machine Learning model that implements the actual prediction) © S-Cube
  • 10. Types of Runtime Data Data that is used to generate predictions can be categorized along 2 dimensions Data Availability in Checkpoint – Available (facts) – Not available, but estimatable (estimates) – Not available (unknown) Type of Runtime Data – External data (not monitorable, needs to be provided by external data sources) – Quality-of-Service information – Domain-specific instance data (e.g., KPIs) © S-Cube
  • 11. Defining Checkpoints (1) Facts: {Customer, OrderedProduct, ...} {AssemblingTime, QoS_ExtSupplier, ...} Data Estimates: {QoS_ExtSupplier, QoS_Warehouse, ...} {QoS_BankingService, ...} Unknown: {InStock, PaymentPrefs, ...} {PaymentPrefs, DeliveryTimeShipment} Order Ship Receive Unavailable Quality Composition RFQ Parts Control Service Produce Assemble Offer Charge Customer Prediction Error Quality / Timeliness Tradeoff Time for Reaction C1 C2 © S-Cube
  • 12. Defining Checkpoints (2) There is a tradeoff to consider when defining checkpoints – Early checkpoints are attractive as they allow for more / better reactions  there simply is more time left – Later checkpoints are attractive as the prediction quality generally improves with time Providers need to identify a checkpoint where … Facts: {Customer, OrderedProduct, ...} {AssemblingTime, QoS_ExtSupplier, ...} Data Estimates: {QoS_ExtSupplier, QoS_Warehouse, ...} {QoS_BankingService, ...} – … the most important Factors of Unknown: {InStock, PaymentPrefs, ...} {PaymentPrefs, DeliveryTimeShipment} Order Ship Influence are already available Receive Unavailable Quality Composition RFQ Parts Control Service – … adaptation is still possible Produce Assemble Offer Charge Customer Prediction Error Quality / Timeliness Tradeoff Time for Reaction C1 C2 © S-Cube
  • 13. Background:Factors of Influence Factors of Influence of business processes are the main factors that lead to performance problems Knowing these factors is essential for targeted optimization of business processes – i.e., if a factor is known to lead to performance problems optimization needs to minimize this factor Research work on identifying Factors of Influence: Wetzstein, Leitner, Rosenberg, Brandic, Dustdar, and Leymann. Monitoring and Analyzing Influential Factors of Business Process Performance. In Proceedings of the 13th IEEE international conference on Enterprise Distributed Object Computing (EDOC09). IEEE Press, Piscataway, NJ, USA, 118-127. Bodenstaff, Wombacher, Reichert, and Jaeger. Monitoring Dependencies for SLAs: The MoDe4SLA Approach. In Proceedings of the 2008 IEEE International Conference on Services Computing (SCC08). IEEE Computer Society, Washington, DC, USA, 21-29. © S-Cube
  • 14. Estimates and External Data One novelty of our approach: missing data can be supplemented via Estimates – Simple example: - the response time of a service that is invoked after the Checkpoint cannot be monitored … - … but of course we can assume that the response time will be similar to the measured average response time of this service -  the average response time can be used as an Estimate for this unknown Factor of Influence Technically, Estimates are not monitored from event streams but provided by dedicated estimator components Similarly, external data can be included via External Data Providers © S-Cube
  • 15. Event-Based Monitoring In order to be used for prediction, runtime data needs to be monitored Event-based monitoring is an often-used idea to implement this Basic principle: Register for and receive some lifecycle events from the service composition and use Complex Event Processing (CEP) to extract monitoring data from raw event data. Can be used to monitor both QoS and domain-specific data © S-Cube
  • 16. Implementing Event-Based Monitoring:Lifecycle Events (1) Many composition engines emit lifecycle (status) events Example 1: Apache ODE WS-BPEL Engine – Currently emits 23 different types of events, including ActivityExecStartEvent, ActivityExecEndEvent, VariableModificationEvent, … – Full list available online Example 2: Windows Workflow Foundation Tracking Service – Emits a smaller number of events by default, but user-defined events can be emitted at any time in a service composition © S-Cube
  • 17. Implementing Event-Based Monitoring:Lifecycle Events (2) These low-level lifecycle events can be aggregated to produce meaningful higher-level metrics (event processing) – Example 1: the execution time of the supplier service is the timestamp of the ‘activity-ended’ event minus the timestamp of the respective ‘activity-started’ event – Example 2: the customer identifier of the customer who put the order can be retrieved from the response message contained in a specific ‘variable-assigned’ event – Example 3: the average failure rate of the shipping service is defined as the number of ‘failed-execution’ events divided by the number of ‘execution-started’ events However, event aggregation middleware necessary to do the necessary calculations on the event streams One possibility: SOA Event Engine implemented in VRESCo © S-Cube
  • 18. Background:The VRESCo SOA Runtime (1) In S-Cube we used the VRESCo SOA Runtime Environment as basis for this research Service Client VRESCo Runtime Environment Query Interface Query VRESCo Client Library Engine Client SOAP Event Daios Notification Database Factory Interface Notification Engine Publishing Interface Control Access Publishing/ Layer ORM invoke QoS Metadata Registry SOAP Monitor Metadata Service Database Interface Management measure Interface Management Service Composition Certificate Interface Composition Store Services Engine © S-Cube
  • 19. Background:The VRESCo SOA Runtime (2) VRESCo is a registry with explicit support for Quality-of- Service and dynamic binding of services Additionally, it’s very easy to build dynamic, loosely coupled service compositions based on Workflow Foundation (WF) technology in VRESCo These WF based compositions emit lifecycle events as Service discussed before using the Client Query Interface VRESCo Runtime Environment Query VRESCo Client Library VRESCo Event Engine, a Engine Client SOAP Event Daios Notification Database Factory Interface Notification complex event processing Engine Publishing Interface Control Access Publishing/ Layer ORM QoS Metadata Registry engine based on NEsper invoke SOAP Monitor Metadata Service Database Interface Management measure Interface Management Service Composition Certificate Interface Composition Store Services Engine © S-Cube
  • 20. Background:The VRESCo SOA Runtime (3) Check the following S-Cube paper for more information on VRESCo: Michlmayr, Rosenberg, Leitner, and Dustdar. End-to-End Support for QoS-Aware Service Selection, Binding, and Mediation in VRESCo. IEEE Transactions on Services Computing 3, 3 (July 2010), 193-205. The VRESCo Event Engine, which has been used as basis for runtime data monitoring, has been introduced in the following earlier publication: Michlmayr, Rosenberg, Leitner, and Dustdar. Advanced Event Processing and Notifications in Service Runtime Environments. In Proceedings of the Second International Conference on Distributed Event-Based Systems (DEBS 08). ACM, New York, NY, USA, 115-125. © S-Cube
  • 21. Architectural Overview of Prediction (1) © S-Cube
  • 22. Architectural Overview of Prediction (2)1. Define Checkpoint as discussed before2. Define important Factors of Influence and monitor them3. Train Machine Learning model from monitored data – For quantitative objectives (e.g., execution time of composition) we use Artificial Neural Networks – For qualitative objectives (e.g., quality of product) we use Decision Trees4. In Checkpoint, generate predictions using trained model © S-Cube
  • 23. Background:Machine Learning (1) Machine Learning is a branch of computer science that deals with algorithms that learn dependencies and relationships from data The basic principle is the idea of generalization, i.e., Machine Learning algorithms aim at finding general rules and relations in concrete data sets Machine Learning is usually a two-step procedure: 1. Firstly, a predictor is learned from a training set 2. Secondly, the predictor can be applied to not-yet-seen data There is a plethora of algorithms available for all kinds of purposes, but in S-Cube we focused on Artificial Neural Networks and Decision Trees © S-Cube
  • 24. Background:Machine Learning (2) Artificial Neural Networks implement regression functions in a bio-inspired way Decision Trees implement classification functions using a simple tree structure © S-Cube
  • 25. How Accurate is my Prediction? (1) Quality of predictor can be evaluated in two ways: – Past data based (how well performs the predictor based on the existing training data?) – Future data based (how well performs the predictor when actually used on not-yet-seen data?) Metric for evaluation based on past data: – Correlation coefficient of predictions and measured values in the training set © S-Cube
  • 26. How Accurate is my Prediction? (2) Unfortunately, this metric is prone to overfitting – But still useful to judge the performance of a trained predictor before it has been used the first time During runtime it is better compare actual predictions with the outcome of instances – Mean Prediction Error (average difference between predicted value and actual value) – Prediction Error Standard Deviation (is the error relatively constant, or are there many outliers?) © S-Cube
  • 27. VRESCo Open Source Implementation The prototype used to do these experiments has been developed as part of the VRESCo project – Can be downloaded from Sourceforge The Sourceforge package also includes the assembling case study – However, setup is not very user-friendly – Get in touch with us if interested in reproducing the case study Uses WEKA Machine Learning toolkit to implement the actual prediction algorithms © S-Cube
  • 28. Learning Package Overview Problem Description Event-Based Runtime Prediction Discussion Conclusions © S-Cube
  • 29. Now Let’s Check How AccuratePredictions are (1) C1 C2 C3 C4 C5 [everything available] Get Payment Charge Prefs Customer Receive Select Order Supplier Check Order From Stock [no supplier Supplier 1 available] Ship Order Order From Supplier 2 Cancel Order 18000 16076 16000 14000 Avg. Error [ms] 12000 10000 Mean Prediction Error 8000 in Checkpoints 6000 4000 2158 2000 1328 989 806 0 70 00 © S-Cube
  • 30. Now Let’s Check How AccuratePredictions are (2) Prediction error is decreasing with time In C2, the error is already reasonably small Hence, C2 may be an C1 C2 [everything available] C3 C4 Get Payment Prefs Charge Customer C5 Receive interesting candidate for a Select Order Supplier Check checkpoint in this case Stock [no supplier available] Order From Supplier 1 Ship Order Order From Supplier 2 Cancel Order 18000 16076 16000 14000 Avg. Error [ms] 12000 10000 Mean Prediction Error 8000 in Checkpoints 6000 4000 2158 2000 1328 989 806 0 70 00 60 00 m © S-Cube [s ] 5030 50 00
  • 31. Some Important Earlier Work We were not the first ones to have similar ideas Important earlier work includes: Sahai, Machiraju, Sayal, van Moorsel, and Casati. Automated SLA Monitoring for Web Services. In Proceedings of the 13th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management (DSOM 2002). Springer-Verlag, Berlin, Heidelberg, 28-41. Zeng, Lingenfelder, Lei, and Chang. Event-Driven Quality of Service Prediction. In Proceedings of the 6th International Conference on Service-Oriented Computing (ICSOC 08), Springer-Verlag, Berlin, Heidelberg, 147-161. © S-Cube
  • 32. Main Advances Over Earlier Work The S-Cube approach to prediction based on event logs improves on earlier work in some important aspects: – Estimations of missing data can be incorporated via Estimates – External data can be incorporated via External Data Providers – Many different algorithms can be used for prediction - Courtesy of the WEKA backend – Prototype implementation available as open source software - Part of the VRESCo project © S-Cube
  • 33. Discussion - Advantages Event log and machine learning based prediction of SLA violations has a number of clear advantages … – Simplicity – the principle approach is easy to understand – Openness – the approach is not limited to event logs, but all kinds of knowledge and data can be included in the prediction – Proven in the real world – machine learning is by now a proven technique that has been successfully applied in many areas – Generality – the approach works both for quantitative and qualitative objectives – Efficiency – even though training of the machine learning models takes some time, generating predictions is very fast © S-Cube
  • 34. Discussion - Disadvantages … but of course the approach also has some disadvantages. – Bootstrapping problem – the approach assumes that some recorded historical event logs are available for training – Necessary domain knowledge – in order to define checkpoints some domain knowledge is necessary – Availability of monitoring data – one of the basic assumptions of the approach is that all necessary data can be monitored (if this is not the case the approach cannot be used) © S-Cube
  • 35. Learning Package Overview Problem Description Event-Based Runtime Prediction Discussion Conclusions © S-Cube
  • 36. Summary Machine learning based techniques can be used to predict performance problems in service compositions Steps: 1. Define a checkpoint in the composition 2. Train machine learning model from historical event log 3. Whenever a composition instance passes the checkpoint, use the monitored data of the instance as input for the machine learning based prediction Allows us to quickly predict problems, so that countermeasures can still be applied in time – If the checkpoint is early enough that reaction is still possible … © S-Cube
  • 37. Further S-Cube ReadingLeitner, Wetzstein, Rosenberg, Michlmayr, Dustdar, and Leymann. Runtime Prediction of Service LevelAgreement Violations for Composite Services. In Proceedings of the 2009 International conference onService-Oriented Computing (ICSOC/ServiceWave09), Springer-Verlag, Berlin, Heidelberg, 176-186.Leitner, Michlmayr, Rosenberg, and Dustdar. Monitoring, Prediction and Prevention of SLA Violations inComposite Services. In Proceedings of the 2010 IEEE International Conference on Web Services (ICWS 10).IEEE Computer Society, Washington, DC, USA, 369-376.Wetzstein, Leitner, Rosenberg, Brandic, Dustdar, and Leymann. Monitoring and Analyzing Influential Factorsof Business Process Performance. In Proceedings of the 13th IEEE international conference on EnterpriseDistributed Object Computing(EDOC09). IEEE Press, Piscataway, NJ, USA, 118-127. © S-Cube
  • 38. Acknowledgements The research leading to these results has received funding from the European Community’s Seventh Framework Programme [FP7/2007-2013] under grant agreement 215483 (S-Cube). © S-Cube

×