Despite significant scientific research, systematic performance engineering techniques are still hardly used in industry, as many practitioners rely on ad-hoc performance firefighting. It is still not well understood where more sophisticated performance modeling approaches are appropriate and the maturity of the existing tools and processes can be improved. While there have been several industrial case studies on performance modeling in the last few years, more experience is needed to better understand the constraints in practice and to optimize existing tool-chains. I gave a talk summarizing six years of performance modeling at ABB. In three projects, different approaches to performance modeling were taken, and experiences on the capabilities and limitations of existing tools were gathered. The talk reports on several lessons learned from these projects, for example the need for more efficient performance modeling and the integration of measurement and modeling tools.

1. © ABB Group
January 7, 2015 | Slide 1
6 Years of Performance Modeling
at ABB Corporate Research
Heiko Koziolek, DECRC/I1 Ladenburg, Germany, 2014-11-13

2. My Story Today
© ABB Group
January 7, 2015 | Slide 2

3. 2008: Performance Modeling with Palladio
Overview
1) Measure
System instrumentation
& performance tests
using load drivers
(custom tooling)
2) Model
Component-based
model with annotated
flow charts entered into
Eclipse-based tooling
3) Predict
Running
analytic solvers /
simulators, varying
model parameters to
test different situations
© ABB Group
January 7, 2015 | Slide 3
Model derivation (manually)
Prediction (automatically)

4. 2010: Q-ImPrESS & Industrial Control System
© ABB Group
January 7, 2015 | Slide 4

5. 2010: Q-ImPrESS
Models & Tools
© ABB Group
January 7, 2015 | Slide 5
 SoMoX / Sissy for Reverse Engineering Component Models from C++
 Windows Performance Monitor for Performance Measurement
 Self-implemented C#-Client as load driver
 Q-ImPreSS Workbench for Modelling (meta model similar to Palladio)
 LQN solver / Palladio SimuCom for Performance Prediction
 PerOpteryx for Design Space Exploration
Q-ImPress WorkbenchSoMoX

6. 2010: Q-ImPrESS
Results
© ABB Group
January 7, 2015 | Slide 6
Koziolek, Schlich et al.
An industrial case study on
quality impact prediction for
evolving service-oriented
software.
In Proc. ICSE 2011 SEIP,
pp. 776-785. ACM, May 2011.

7. 2010: Q-ImPrESS
Lessons Learned
 Successes
 Large performance model build with Q-ImPreSS tooling
 Models validated through measurements (<30% error)
 First experiments with design space exploration
 Challenges
 Not enough inputs on new ABB system available,
had to fallback to model older version
predictions for older system are not really actionable
as the older version will not be changed
 Modeling tools disconnected from the tools currently
used during development (e.g., Enterprise Architect)
creating models with the tools from scratch required
high effort
 Static code analysis challenged by Microsoft C++ code
© ABB Group
January 7, 2015 | Slide 7

8. 2012: Performance Modeling for ABB Robotics
© ABB Group
January 7, 2015 | Slide 8

9. 2012: Performance Modeling for ABB Robotics
Models & Tools
 Dynatrace for distributed performance profiling
 Neoload as load driver
 Palladio Workbench for modelling ‚
(all manual no static code analysis)
 LQN/SimuCom for performance prediction
 PerOpteryx for design space exploration© ABB Group
January 7, 2015 | Slide 9
Palladio Workbench
LQN Solver

10. 2012: Performance Modeling for ABB Robotics
Results
© ABB Group
January 7, 2015 | Slide 10
Thijmen de Gooijer, Anton Jansen, Heiko Koziolek, and Anne Koziolek. An industrial case study of performance and cost
design space exploration. In Proc. 3rd Int. Conf. on Performance Engineering (ICPE'12), pp 205-216. ACM, April 2012.

11. 2012: Performance Modeling for ABB Robotics
Lessons Learned
 Successes
 Due to performance fixes based on the measurements
the performance could be improved by 50%
 Roadmap for extending the system cost-effectively was devised
based on the models
 Large-scale industrial case study on design space exploration of
a distributed, component-based system
 ABB Robotics integrated Dynatrace into their development
environment
 Challenges
 Information extraction for the models took a long time, lots of
calibration needed, several assumptions required for models
 Expensive measurement & testing tools (>30K€)
© ABB Group
January 7, 2015 | Slide 11

12. 2014: Automation Cloud
© ABB Group
January 7, 2015 | Slide 12

13. 2014: Automation Cloud
Models & Tools
 Amazon Web Services / Own Cloud Server as test
environment (up to 36 AWS m1.large instances)
 KairosDB, OpenTSDB, Databus as time-series databases
 Apache Cassandra and Hbase as distributed DBMS
 Netflix Priam / Apache Whirr for quick deployment
 Visual Studio Ultimate Web Load Test as load driver
 [No modeling, only benchmarks!]© ABB Group
January 7, 2015 | Slide 13

14. 2014: Automation Cloud
Results
© ABB Group
January 7, 2015 | Slide 14
Limited overload WITH
AWS Autoscaling
Linear scalability
for KairosDB
Avg. Roundtrip Time: 193ms
for 15 customers
ABB Phasor Measurement Unit
used in Power Grids
ABB Smart Meter
Thomas Goldschmidt, Anton Jansen, Heiko Koziolek, Jens Doppelhamer, and Hongyu Pei-Breivold. Scalability
and Robustness of Time-Series Databases for Cloud-Native Monitoring of Industrial Processes. In Proceedings
7th IEEE Int. Conf. on Cloud Computing (IEEE CLOUD 2014) Industry Track. IEEE, July 2014.

15. 2014: Automation Cloud
Lessons Learned
 Successes
 Showed technical feasibility for several scenarios from
industrial automation in a cloud computing environment
 Created benchmarks for time-series databases based
on realistic workloads from ABB products
 Created elasticity metrics and benchmark
 Challenges
 Better testing needed to improve robustness
 Component-based model did not well fit with databases
/ cloud platform (e.g., auto-scaling?)
 Limited insights expected from modeling due to focus
on initial technical feasibility
© ABB Group
January 7, 2015 | Slide 15

16. 2015 Outlook: Collaboration with Uni Würzburg
Automatic Construction of Architectural Perf. Models
© ABB Group
January 7, 2015 | Slide 16
 Kieker + C# adapter / JNBridge for distributed profiling
 LibReDe for resource demand estimation
 .NET Bookstore / Pet Shop (C#) for testing, later ABB system
 Palladio / Descartes / PerOpteryx for modeling / prediction

17. 6 Years of Performance Modeling at ABB
Conclusions
 Performance modeling has matured over the last 6 years
 But: to get wider adoption
lower costs and higher benefits
are required.
© ABB Group
January 7, 2015 | Slide 17

18. 6 Years of Performance Modeling at ABB
Future Work
 Future work for lower costs
 Better integration between measurement and modeling tools
 Faster modeling via more convenient software tools
 Faster modeling via reusable model libraries
 Future work for higher benefits
 More performance questions to be answered
 Decision support and incorporation of heuristics
 Better integration into existing development processes & tools
© ABB Group
January 7, 2015 | Slide 18