Mastering Software Variability for Innovation and Science
•
0 likes•52 views
Software has eaten the world and will continue to impact society, spanning numerous application domains, including the way we're doing science and acquiring knowledge. Software variability is key since developers, engineers, entrepreneurs, scientists can explore different hypothesis, methods, and custom products to hopefully fit a diversity of needs and usage. In this talk, I will first illustrate the importance of software variability using different concrete examples. I will then show that, though highly desirable, software variability also introduces an enormous complexity due to the combinatorial explosion of possible variants. For example, 10^6000 variants of the Linux kernel may be build, much more than the estimated number of atoms in the universe (10^80). Hence, the big picture and challenge for the audience: any organization capable of mastering software variability will have a competitive advantage. Summer School EIT Digital. 5th July Rennes 2022
Recommended
More Related Content
Similar to Mastering Software Variability for Innovation and Science
Similar to Mastering Software Variability for Innovation and Science (20)
More from University of Rennes, INSA Rennes, Inria/IRISA, CNRS
More from University of Rennes, INSA Rennes, Inria/IRISA, CNRS (20)
Recently uploaded
Recently uploaded (20)
Mastering Software Variability for Innovation and Science
- 1. Mastering Software Variability for Innovation and Science Mathieu Acher @acherm mathieuacher.com #SciencesDuLogiciel 1
- 2. Summer School EIT Digital. 5th July Rennes 2022 Abstract : Software has eaten the world and will continue to impact society, spanning numerous application domains, including the way we're doing science and acquiring knowledge. Software variability is key since developers, engineers, entrepreneurs, scientists can explore different hypothesis, methods, and custom products to hopefully fit a diversity of needs and usage. In this talk, I will first illustrate the importance of software variability using different concrete examples. I will then show that, though highly desirable, software variability also introduces an enormous complexity due to the combinatorial explosion of possible variants. For example, 10^6000 variants of the Linux kernel may be build, much more than the estimated number of atoms in the universe (10^80). Hence, the big picture and challenge for the audience: any organization capable of mastering software variability will have a competitive advantage. 2
- 3. Software variants are eating the world Software variability for supporting innovation and science Improving, Adapting, Exploring, Creating cutting-edge products Software science and software variability Challenges ahead! Agenda 3
- 4. Software variants are eating the world 4
- 5. Software is eating science 5
- 6. Science now depends on software and its engineering 6 design of mathematical model mining and analysis of data executions of large simulations problem solving executable paper from a set of scripts to automate the deployment to… a comprehensive system containing several features that help researchers exploring various hypotheses
- 7. 7 from a set of scripts to automate the deployment to… a comprehensive system containing several features that help researchers exploring various hypotheses multi-million line of code base multi-dependencies multi-systems multi-layer multi-version multi-person multi-variant Science now depends on software and its engineering
- 8. 8 Dealing with software collapse: software stops working eventually Konrad Hinsen 2019 Configuration failures represent one of the most common types of software failures Sayagh et al. TSE 2018 multi-million line of code base multi-dependencies multi-systems multi-layer multi-version multi-person multi-variant Science now depends on software and its engineering
- 11. Linux Kernel 11
- 12. 15,000+ options thousands of compiler flags and compile-time options dozens of preferences 100+ command-line parameters 1000+ feature toggles 12 hardware variability Deep software variability
- 14. 14
- 15. “Reverse Engineering Web Configurators” Ebrahim Khalil Abbasi, Mathieu Acher, Patrick Heymans, and Anthony Cleve. In 17th European Conference on Software Maintenance and Reengineering (CSMR'14) 15
- 16. Software variants are eating the world Software variability for supporting innovation and science Improving, Adapting, Exploring, Creating cutting-edge products Software science and software variability Challenges ahead! Agenda 16
- 17. Too many knobs and variability? default configurations are suboptimal Xu et al. FSE 2015 17
- 18. Best configuration is 480 times better than Worst configuration Only by tweaking 2 options out of 200 in Apache Storm observed ~100% change in latency Best Worst Perhaps… but missing opportunities with software variability! Jamshidi et al. An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems MASCOT’16 18
- 19. Best configuration is 480 times better than Worst configuration Only by tweaking 2 options out of 200 in Apache Storm observed ~100% change in latency Best Worst Mastering software variability for delivering faster while consuming less energy Jamshidi et al. An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems MASCOT’16 19
- 20. 20 Mathieu Acher, Benoit Baudry, Olivier Barais, Jean-Marc Jézéquel: Customization and 3D printing: a challenging playground for software product lines. SPLC 2014: 142-146
- 21. 21 Mastering software variability for avoiding configuration errors and assisting users in finding the right products Benoit Amand, Maxime Cordy, Patrick Heymans, Mathieu Acher, Paul Temple, Jean-Marc Jézéquel Towards Learning-Aided Configuration in 3D Printing: Feasibility Study and Application to Defect Prediction VaMoS 2019
- 22. ● Web-apps generator ○ Spring-Boot ○ Bootstrap / AngularJS ○ Open Source ● Yeoman ○ Bower, npm ○ yo ● Used all over the world ○ Large companies (HBO, Google, Adobe)1 ○ Independent developers ○ Our students ● JHipster 3.6.1, 18 Aug 2016 ○ 6000+ stars ○ 118 releases ○ 300+ contributors 22 Mastering software variability for adapting to your technological constraints or know-how A. Halin, A. Nuttinck, M. Acher, X. Devroey, G. Perrouin and B. Baudry, “Test them all, is it worth it? Assessing configuration sampling on the JHipster Web develop- ment stack”, Empirical Software Engineering (ESE)
- 23. José A. Galindo, Mauricio Alférez, Mathieu Acher, Benoit Baudry, David Benavides: A variability-based testing approach for synthesizing video sequences. ISSTA 2014 Mastering software variability for synthesizing a diversity of input data either for training or testing! 23
- 24. Mastering software variability for exploring different variants (= PDF papers) that fit your requirements Mathieu Acher, Paul Temple, Jean-Marc Jézéquel, José Ángel Galindo Duarte, Jabier Martinez, Tewfik Ziadi VaryLaTeX: Learning Paper Variants That Meet Constraints VaMoS 2020 24
- 25. ????? Mastering software variability for predicting performance of a system (without running it!) J. Alves Pereira, M. Acher, H. Martin and J.-M. Jézéquel, “Sampling Effect on Performance Prediction of Configurable Systems: A Case Study” ICPE’20, Best paper award 25
- 26. compile-time options L. Lesoil, M. Acher, X. Tërnava, A. Blouin and J.-M. Jézéquel “The Interplay of Compile-time and Run-time Options for Performance Prediction” in SPLC ’21 Mastering software variability for predicting performance of a system (without running it!) 26
- 27. 27
- 28. [Xu et al., FSE 2015] [Passos et al., TSE 2016] Huge variability space: impossible to execute, obs and label all configurations/variants in practice! 33 options = more configurations than humans on earth 320 options = more configurations than estimated atoms in the unive 28
- 29. Whole Population of Configurations Performance Prediction Training Sample Performance Measurements Prediction Model J. Alves Pereira, H. Martin, M. Acher, J.-M. Jézéquel, G. Botterweck and A. Ventresque “Learning Software Configuration Spaces: A Systematic Literature Review” JSS, 2021 Sampling, Measuring, Learning 29
- 30. Huge variability space: impossible to execute, observe, and label all configurations in practice Key idea: learning out of a sample of configurations that is hopefully small/representative of the whole space J. Alves Pereira, H. Martin, M. Acher, J.-M. Jézéquel, G. Botterweck and A. Ventresque “Learning Software Configuration Spaces: A Systematic Literature Review” JSS, 2021 30
- 31. 31
- 32. 15,000+ options Linux 5.2.8, arm (% of types’ options) 39000 26000 ≈106000 variants (without constraints) 32
- 34. Configurable software system Configurations Variants Quantitative property (eg related to performance, security, energy consumption) 176.8Mb Linux kernel .config (compile-time/Kconfig) Kernel variants (binaries) binary size 34
- 35. Configurable software system Configurations Variants Quantitative property (eg related to performance, security, energy consumption) 16.1Mb Linux kernel .config (compile-time/Kconfig) Kernel variants (binaries) binary size 35
- 36. Configurable software system Configurations Variants Quantitative property (eg related to performance, security, energy consumption) 176.8Mb Linux kernel .config (compile-time/Kconfig) Kernel variants (binaries) binary size 16.1Mb 77.2Mb 36
- 37. (size of vmlinux) Max: 1,698.14Mb Min: 7Mb (tinyconfig) 37
- 38. Configurable software system Configurations Variants Quantitative property (eg related to performance, security, energy consumption) Linux kernel .config (compile-time/Kconfig) Kernel variants (binaries) binary size ? 38
- 39. Challenge 1: you cannot build ≈106000 configurations; sampling and learning to the rescue but how? and what about cost? 7.1Mb 176.8Mb ? Variability in space 39
- 40. A challenging case ● Linear-based algorithms : high error rate (it’s not additive!) ● Polynomial regression & performance-influence model : Out Of Memory (too much interactions and not designed for 9K+ options) ● Tree-based algorithms & neural networks: low error rate Mean Absolute Percentage Error (MAPE): the lower the better 40 Mathieu Acher, Hugo Martin, Juliana Alves Pereira, Arnaud Blouin, Jean-Marc Jézéquel, Djamel Eddine Khelladi, Luc Lesoil, and Olivier Barais. Learning Very Large Configuration Spaces: What Matters for Linux Kernel Sizes (2019) https://hal.inria.fr/hal-02314830 N : percentage of the dataset used to training (95K in total) for Linux 4.13.3
- 41. Challenge 2: Linux evolves; (heterogeneous) transfer learning! 7.1Mb 176.8Mb ? v4.13 v5.8 3 years later… ? ? ? Variability in space …Variability in time… 41
- 42. Instead of learning from scratch or reusing “as is”, we propose to transfer (adapt) a prediction model. A problem overlooked in the literature is that the feature spaces differ among versions since options are added/removed. We propose TEAMS for transfer evolution-aware model shifting. Key results: Rapid degradation of prediction models due to evolution (up to 32% prediction errors) Effective transfer learning with TEAMS (reaching same accuracy 3 years later + SOTA) Possible to learn colossal configuration spaces such as the Linux kernel one across variants and versions 42
- 43. Can we reuse a prediction model among versions? No. Accuracy quickly decreases: ○ 4.13: 6% ○ 4.15: 20% ○ 5.7: 35% 3 43 Insights about evolution of (important) options in the TSE article!
- 44. 4.13 version (sep 2017): 6%. What about evolution? Can we reuse the 4.13 Linux prediction model? No, accuracy quickly decreases: 4.15 (5 months after): 20%; 5.7 (3 years after): 35% 3 44
- 45. Transfer learning to the rescue ● Mission Impossible: Saving variability knowledge and prediction model 4.13 (15K hours of computation) ● Heterogeneous transfer learning: the feature space is different ● TEAMS: transfer evolution-aware model shifting 5 45 H. Martin, M. Acher, J. A. Pereira, L. Lesoil, J. Jézéquel and D. E. Khelladi, “Transfer learning across variants and versions: The case of linux kernel size” Transactions on Software Engineering (TSE), 2021 3 45
- 46. Published at IEEE Transactions on Software Engineering (TSE) in 2021 Preprint: https://hal.inria.fr/hal-03358817 46
- 47. Software variants are eating the world Software variability for supporting innovation and science Improving, Adapting, Exploring, Creating cutting-edge products Software science and software variability Challenges ahead! Agenda 47
- 48. 48 from a set of scripts to automate the deployment to… a comprehensive system containing several features that help researchers exploring various hypotheses Software variability is key to scientific, software-based experiments
- 49. (computational) science 49 from a set of scripts to automate the deployment to… a comprehensive system containing several features that help researchers exploring various hypotheses Software variability is a threat to scientific, software-based experiments
- 50. “Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.” Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015 The implementation of mathematical functions manipulating single-precision floating-point numbers in libmath has evolved during the last years, leading to numerical differences in computational results. While these differences have little or no impact on simple analysis pipelines such as brain extraction and cortical tissue classification, their accumulation creates important differences in longer pipelines such as the subcortical tissue classification, RSfMRI analysis, and cortical thickness extraction. 50
- 51. Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? A study involving eight institutions and seven different supercomputers in Europe is currently ongoing with EC-Earth. This ongoing study aims to do the following: ● evaluate different computational environments that are used in collaboration to produce CMIP6 experiments (can we safely create large ensembles composed of subsets that emanate from different partners of the consortium?); ● detect if the same CMIP6 configuration is replicable among platforms of the EC-Earth consortium (that is, can we safely exchange restarts with EC-Earth partners in order to initialize simulations and to avoid long spin-ups?); and ● systematically evaluate the impact of different compilation flag options (that is, what is the highest acceptable level of optimization that will not break the replicability of EC-Earth for a given environment?). 51
- 52. Joelle Pineau “Building Reproducible, Reusable, and Robust Machine Learning Software” ICSE’19 keynote “[...] results can be brittle to even minor perturbations in the domain or experimental procedure” What is the magnitude of the effect hyperparameter settings can have on baseline performance? How does the choice of network architecture for the policy and value function approximation affect performance? How can the reward scale affect results? Can random seeds drastically alter performance? How do the environment properties affect variability in reported RL algorithm performance? Are commonly used baseline implementations comparable? 52
- 53. (computational) science 53 from a set of scripts to automate the deployment to… a comprehensive system containing several features that help researchers exploring various hypotheses Software variability is an opportunity to robustify and augment scientific knowledge (and thus for innovation)
- 54. Developers, engineers, entrepreneurs, scientists should have the super-power of exploring variants with software (new “configurations”, “products”, “ideas”, “hypotheses”...) 54
- 55. Challenge #0 Mastering deep software variability Perf can be either execution time, energy consumption, accuracy, security, etc. (or a combination thereof) 55
- 56. Challenge #1 Reducing the cost of exploring the variability spaces/layers Many directions here ● learning ○ many algorithms/techniques with tradeoffs interpretability/accuracy ○ transfer learning (instead of learning from scratch) ● sampling strategies ○ uniform random sampling? t-wise? distance-based? … ○ sample of hardware? input data? ● incremental build of configurations ● white-box approaches ● … 56
- 57. Challenge #2 Modelling deep variability ● Abstractions are definitely needed to… ○ reason about logical constraints and interactions ○ integrate domain knowledge ○ synthesize domain knowledge ○ automate and guide the exploration of variants ○ scope and prioritize experiments ● Challenges: ○ Multiple systems, layers, concerns ○ Different kinds of variability: technical vs domain, accidental vs essential, implicit vs explicit… when to stop modelling? ○ reverse engineering 57
- 58. Challenge #3 Robustness (trustworthiness) of scientific results to sources of variability There are many examples of sources of variations (and non-robust results). Threats for science and innovation! How to verify the effect of sources of variations on the robustness of given conclusions? ● actionable metrics? ● methodology? (eg when to stop?) ● variability can actually be leveraged to augment confidence 58
- 59. Conclusion Science for software engineering and variability needed to support innovation and science 59
- 60. (BEYOND x264) Empirical and fundamental question HOW DOES DEEP SOFTWARE VARIABILITY MANIFEST IN THE WILD? SOFTWARE SCIENTISTS should OBSERVE the JUNGLE/ GALAXY! Age # Cores GPU Variant Compil. Version Version Option Distrib. Size Length Res. Hardware Operating System Software Input Data 60
- 61. Deep Software Variability 4 challenges and opportunities Luc Lesoil, Mathieu Acher, Arnaud Blouin, Jean-Marc Jézéquel: Deep Software Variability: Towards Handling Cross-Layer Configuration. VaMoS 2021: 10:1-10:8 61
- 62. Identify the influential layers 1 Age # Cores GPU Variant Compil. Version Version Option Distrib. Size Length Res. Hardware Operating System Software Input Data Problem ≠ layers, ≠ importances on performances Challenge Estimate their effects Opportunity Leverage the useful variability layers & variables 62
- 63. 2 Test & Benchmark environments 0.152.2854 0.155.2917 Problem Combinatorial explosion and cost Challenge Build a representative, cheap set of environments Opportunity dimensionality reduction Hardware Operating System Software Input Data 63
- 64. Transfer performances across environments 10.4 20.04 Dell latitude 7400 Raspberry Pi 4 model B A B ? Problem Options’ importances change with environments Challenge Transfer performances across environments Opportunity Reduce cost of measure 3 64
- 65. Cross-Layer Tuning Age # Cores GPU Variant Compil. Version Version Option Distrib. Size Length Res. Hardware Operating System Software Input Data 4 Problem (Negative) interactions of layers Challenge Find & fix values to improve performances Opportunity Specialize the environment for a use case Bug Perf. ↗ Perf. ↘ 65
- 66. CHALLENGES for Deep Software Variability Identify the influential layers Test & Benchmark environments Transfer performances across environments Cross-Layer Tuning 66
- 67. Wrap-up Deep software variability is a thing… miracle and smooth fit of variability layers? or subtle interactions among layers? software scientists should systematically study the phenomenon Many challenges and opportunities cross-layer tuning at affordable cost configuration knowledge that generalizes to any context and usage Dimensionality reduction and transfer learning 67
- 68. Impacts of deep software variability Users/developers/scientists: missed opportunities due to (accidental) variability/complexity Deep software variability can be seen as a threat to knowledge/validity Claim: Deep software variability is a threat to scientific, software-based experiments Example #1: in the results of neuroimaging studies ● applications of different analysis pipelines (Krefting et al. 2011) ● alterations in software version (Glatard et al. 2015) ● changes in operating system (Gronenschild2012 et al.) have both shown to cause variation (up to the point it can change the conclusions). Example #2: on a modest scale, our ICPE 2020 (Pereira et al.) showed that a variation in inputs could change the conclusion about the effectiveness of a sampling strategy for the same software system. Example #3: I’m reading Zakaria Ournani et al. “Taming Energy Consumption Variations In Systems Benchmarking” ICPE 2020 as an excellent inquiry to control deep variability factors Example #4: Similar observations have been made in the machine learning community (Henderson et al. 2018) version runtime options OS 68
- 69. Impacts of deep software variability Users/developers/scientists: missed opportunities due to (accidental) complexity Deep software variability can be seen as a threat to knowledge/validity Claim: Deep software variability is a threat to scientific, software-based experiments I propose an exercise that might be slightly disturbing: Does deep software variability affect (your|a known) previous scientific, software-based studies? (remark: it’s mainly how we’ve investigated deep software variability so far… either as a threat to our own experiments or as a threat identified in papers) 69
- 70. Applications of different analysis pipelines, alterations in software version, and even changes in operating system have both shown to cause variation in the results of a neuroimaging study. Example: Joelle Pineau “Building Reproducible, Reusable, and Robust Machine Learning Software” ICSE’19 keynote “[...] results can be brittle to even minor perturbations in the domain or experimental procedure” Example: DEEP SOFTWARE VARIABILITY DEEP SOFTWARE VARIABILITY Deep software variability is a threat to scientific, software-based experiments 70
- 72. 15,000+ options Linux 5.2.8, arm (% of types’ options) 39000 26000 ≈106000 variants (without constraints) 72