Q-ImPrESS

756 views

Published on

Systematic decision support for architectural design decisions is a major concern for software architects of evolving service-oriented systems. In practice, architects often analyse the expected performance and reliability of design alternatives based on prototypes or former experience. Model-driven prediction methods claim to uncover the tradeoffs between different alternatives quantitatively while being more cost-effective and less error-prone. However, they often suffer from weak tool support and focus on single quality attributes. Furthermore, there is limited evidence on their effectiveness
based on documented industrial case studies. Thus, we have applied a novel, model-driven prediction method called Q-ImPrESS on a large-scale process control system consisting of several million lines of code from the automation domain to evaluate its evolution scenarios. This presentation reports our experiences with the method and lessons learned. Benefits of Q-ImPrESS are the good architectural decision support and comprehensive tool framework, while one drawback is the time-consuming data
collection.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
756
On SlideShare
0
From Embeds
0
Number of Embeds
75
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 3 MLOC C++, COM, ATL9 subsystems, >100 componentsmanaging industrial process (e.g., power generation, paper production, oil and gas refining, etc.)distributed system, controllers, servers, networks, field devicesoperator workplace for controlling the process: montoring sensor readings, manipulating actuators
  • units obfuscated for confidentiality reasonssubsystem 8 has highest failure probabilitysubsystem 1 has highest sensitivity to system reliabilitysubsystem 6 is used by many subsystems, but only limited contribution to system reliability
  • Q-ImPrESS

    1. 1. © ABB Group January 29, 2015 | Slide 1 Q-ImPrESS Quality Impact Prediction for Evolving Service-oriented Software Heiko Koziolek, DECRC/I1 Industrial Software Technologies, 2010-11-26
    2. 2. © ABB Group January 29, 2015 | Slide 2 Agenda Context & Motivation ABB Demonstrator Applying Q-ImPrESS Lessons Learned
    3. 3. EU Project Q-ImPrESS Context © ABB Group January 29, 2015 | Slide 3 First Discussions 01/2008 Demonstrators finished 12/201004/2009 Tools available for testing 12/2008 Requirements finished 12/2009 First Iteration Demonstrators Consortium members Q-ImPrESS tools Ericsson demonstrator ABB demonstrator
    4. 4. Industrial automation: Process Control Systems © ABB Group January 29, 2015 | Slide 4
    5. 5. Motivation Software development at ABB  Continuous evolution of ABB software systems  New requirements, technologies, failure reports Software maintenance and evolution are a large cost factor for ABB software development  Current practice  Design decisions for evolutions based on experience  Prototyping for new technologies, performance impacts  Limited systematic tradeoff analyses Apply model-based prediction methods for systematic tradeoff analyses to save costs and achieve higher quality? © ABB Group January 29, 2015 | Slide 5
    6. 6. Q-ImPrESS Method Overview Eclipse Modeling Framework Q-ImPrESS Workbench Backbone Layered Queueing Networks or Palladio/SimuCOM Discrete Time Markov Chains KAMP cost calculator QVT Model Transformations
    7. 7. ABB Demonstrator: Process control system Topology © ABB Group January 29, 2015 | Slide 7 Plant / Office Network Network Isolation Device Remote Workplaces Firewall Internet Remote Workplaces Redundant Network Workplaces Controllers Servers Fieldbus Remote I/O and Field devices Focus on the server-side software
    8. 8. Reverse Engineering in Q-ImPrESS Process/Method © ABB Group January 29, 2015 | Slide 8 SoMoX3 Q-ImPrESS Model C++ Source 1 Structural Investigation of Software Systems 3 Software Model eXtractor 2 Generalized Abstract Syntax Tree SISSy1 GAST2
    9. 9. Reverse Engineering in Q-ImPrESS Sample Result for 50KLOC subsystem © ABB Group January 29, 2015 | Slide 9
    10. 10.  SISSy parser not robust against Microsoft C++ code  Configuring of the metrics intransparent  Reference decomposition essential  Limited scalability, unclear if faster than manual modelling  Static analysis of limited use for QoS prediction Reverse Engineering Results from ABB code Lines of Code Time for parsing, clustering Percentage of reconstructed higher level components Qualitative evaluation (--, -, o, +, ++) ABB Component 1 10 0.5h / 1h 100% + ABB Component 2 250 4h / 16h 0% -- ABB Component 3 3300 >60h / ? n/a -- © ABB Group January 29, 2015 | Slide 10
    11. 11. Manual Modeling Process/Method Analyzed existing architectural documentation Created test- bed (2 quad- cores, emulated controllers) Installed and configured the system (>100h) Identified 4 critical usage scenarios Configured load drivers with typical workloads Decided for abstr. level (processes = components) Recorded component transitions Wrote script to derive transition probabilities Discussed validity with architects and developers © ABB Group January 29, 2015 | Slide 11
    12. 12. Manual Modeling Q-ImPrESS model of the ABB process control system © ABB Group January 29, 2015 | Slide 12
    13. 13. Performance Prediction Process Applied Service Demand Law (S = X/U) Measured X, U for each process (PerfMon) Determined 20 CPU and HD resource demands Parametrized resource demands (linear regr.) Abstracted memory, network, virtualization Integrated demands into Q-ImPrESS model Performed predictions (SimuCom, LQN) © ABB Group January 29, 2015 | Slide 13
    14. 14. Performance Prediction Sample predictions for different allocation scenarios © ABB Group January 29, 2015 | Slide 14 Heiko Koziolek. Performance Prediction for an Industrial Control System. Submitted to ACM/SPEC International Conference on Performance Engineering (ICPE‘11)
    15. 15.  Achieved prediction error below 30 percent  Data collection consumed more time than expected  Support for service-oriented systems limited  Easy to analyse different evolution scenarios  Many bottlenecks below the architectural level Performance Prediction Results Workload PerfMon Measured SimuCom Prediction Error (%) LQNS Prediction Error (%) 30 17.146 12.467 27.288 12.464 27.305 60 26.681 22.366 16.174 22.343 16.260 90 31.902 32.347 1.395 32.322 1.317 120 39.016 42.432 8.754 42.329 8.490 150 51.929 51.943 0.027 51.760 0.326 © ABB Group January 29, 2015 | Slide 15
    16. 16. Reliability Prediction Data Collection Acquired access to bug tracking system Selected growth model from IEEE 1633-2008 Performed curve fitting on bug report dates (CASRE) Determined failure rates per subsystem Validated failure rates against code metrics Annotated components in Q-ImPrESS model Performed reliability predictions (PRISM) © ABB Group January 29, 2015 | Slide 16
    17. 17. Reliability Prediction Sample sensitivity analysis © ABB Group January 29, 2015 | Slide 17 Heiko Koziolek, Bastian Schlich, and Carlos Bilich. A Large-Scale Industrial Case Study on Architecture-based Software Reliability Analysis. Proc. 21st IEEE International Symposium on Software Reliability Engineering (ISSRE'10)
    18. 18. Reliability Prediction Results © ABB Group January 29, 2015 | Slide 18  Possible use of results for future test budget distribution  Abstraction level too high: no hardware, no concurrency  Data collection for lower abstraction levels difficult  Many statistical assumptions (e.g., Markovian control flow)  Validation inherently difficult: would take years to get data  Difficult to establish trust into the results  Data collection method not applicable for new systems  Limited step-by-step guidance and best practice known  Limited amount of case studies in literature
    19. 19. Tradeoff Analysis Design space exploration with PerOpteryx © ABB Group January 29, 2015 | Slide 19  Manually specified degrees of freedom in the model (processor speeds, component allocations, COTS)  PerOpteryx applied an evolutionary algorithm (NSGA II) to search the space spanned by the degrees of freedom  Determines optimal performance/reliability/costs tradeoffs  Could be applied for future sizing of the system Anne Martens, Heiko Koziolek, Steffen Becker, and Ralf Reussner. Automatically improve software architecture models for performance, reliability, and cost using evolutionary algorithms. Proc. ACM/SPEC International Conference on Performance Engineering (ICPE'10)
    20. 20.  Analytical Hierachy Process (from decision support)  Requires manual pairwise comparison of candidates  Applied on 4 alternatives, specified 24 weights  Alternative 3 best because of the preference for reliability  Method not scalable: e.g., 180 comparisons for 10 alt. Tradeoff Analysis Weighing Alternatives with AHP Wizard © ABB Group January 29, 2015 | Slide 20 Leo Hatvani, Anton Jansen, Cristina Seceleanu and Paul Pettersson. Integrated Tool for Trade-off Analysis of Quality-of-Service Attributes Proc. 2nd Int. Workshop on the Quality of Service-oriented Software Systems (QUAOSS’10)
    21. 21. Lessons Learned Experience with Q-ImPrESS method and tools  Integrated approach with comprehensive tool support  Limited integration with other tools and methods  Cost-effective only for large systems  Limited expressiveness of the Q-ImPrESS meta model  Missing data collection support  Complexity of evolution scenarios determines accuracy  Iterative modelling and strong goal-orientation required © ABB Group January 29, 2015 | Slide 21
    22. 22. Lessons Learned Cost estimations in person hours  Cost estimation (in person hours) based on our experience  Best case: input data or test-bed readily available  Worst case: unknown system, new data collection method  We actually needed more effort than the worst case! © ABB Group January 29, 2015 | Slide 22
    23. 23. Conclusions  Q-ImPrESS reverse engineering  Parser not robust for MS C++, clustering intransparent  Q-ImPrESS modeling tools  Well-integrated, graphical editors  Q-ImPrESS performance prediction  Accurate results possible  Q-ImPrESS reliability prediciton  Many assumptions, still primitive models  More research for efficient data collection needed  Internal ABB follow-up planned © ABB Group January 29, 2015 | Slide 23
    24. 24. © ABB Group January 29, 2015 | Slide 24

    ×