A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments
•
0 likes•702 views
Software performance testing is an important activity to ensure quality in continuous software development environments. Current performance testing approaches are mostly based on scripting languages and framework where users implement, in a procedural way, the performance tests they want to issue to the system under test. However, existing solutions lack support for explicitly declaring the performance test goals and intents. Thus, while it is possible to express how to execute a performance test, its purpose and applicability context remain implicitly described. In this work, we propose a declarative domain specific language (DSL) for software performance testing and a model-driven framework that can be programmed using the mentioned language and drive the end-to-end process of executing performance tests. Users of the DSL and the framework can specify their performance intents by relying on a powerful goal-oriented language, where standard (e.g., load tests) and more advanced (e.g., stability boundary detection and configuration tests) performance tests can be specified starting from templates. The DSL and the framework have been designed to be integrated into a continuous software development process and validated through extensive use cases that illustrate the expressiveness of the goal-oriented language, and the powerful control it enables on the end-to-end performance test execution to determine how to reach the declared intent. My talk from The 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018). Cite us: https://dl.acm.org/citation.cfm?id=3184417
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments
Similar to A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments (20)
More from Vincenzo Ferme
More from Vincenzo Ferme (10)
Recently uploaded
Recently uploaded (20)
A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments
- 1. Università della Svizzera italiana A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments Vincenzo Ferme Software Institute Faculty of Informatics USI Lugano, Switzerland Cesare Pautasso
- 2. 2 Repo Continuous Software Development Environments Developers, Testers, Architects
- 3. 2 CI ServerRepo Continuous Software Development Environments Developers, Testers, Architects
- 4. 2 CI ServerRepo Continuous Software Development Environments Developers, Testers, Architects
- 5. 2 CI ServerRepo Continuous Software Development Environments Developers, Testers, Architects Continuous Changes
- 6. 2 CI ServerRepo Continuous Software Development Environments Developers, Testers, Architects Continuous Changes Continuous Test Execution
- 7. 3 CI ServerRepo Continuous Software Development Environments Developers, Testers, Architects Continuous Changes Continuous Test Execution Load Test, Spike Test
- 8. 3 CI ServerRepo Continuous Software Development Environments Developers, Testers, Architects Continuous Changes Continuous Test Execution Capacity Test Configuration Test Load Test, Spike Test
- 9. State of Practice 4 Script/UI Based End-to-end Tests against REST APIs
- 10. State of Practice 4 Script/UI Based Integrated in CI Pipelines End-to-end Tests against REST APIs
- 11. State of Practice 4 Script/UI Based Integrated in CI Pipelines Performance Tests Execution Automation Google Vizier AutoPerf DataMill CloudPerf End-to-end Tests against REST APIs
- 12. Define a Configuration Test 5 Test Definition
- 13. Define a Configuration Test 5 Test Definition Configuration Parameters Change Simulated Users
- 14. Define a Configuration Test 5 Test Definition Configuration Parameters Change Simulated Users System Under Test 3rd-party tools
- 15. Define a Configuration Test 5 Test Definition Configuration Parameters Change Simulated Users System Under Test 3rd-party tools Test Execution, Data Collection, Metrics Computation
- 16. Define a Configuration Test 5 Test Definition Configuration Parameters Change Simulated Users System Under Test 3rd-party tools Test Execution, Data Collection, Metrics Computation Test Goals are not Explicit in the Definition
- 17. Define a Configuration Test 5 Test Definition Configuration Parameters Change Simulated Users System Under Test 3rd-party tools Test Execution, Data Collection, Metrics Computation N o Sem antic V alidation (Runtim e Errors) Test Goals are not Explicit in the Definition
- 18. Define a Configuration Test 5 Test Definition Configuration Parameters Change Simulated Users System Under Test 3rd-party tools Test Execution, Data Collection, Metrics Computation N o Sem antic V alidation (Runtim e Errors) Scarce Visibility and Control on the Test Execution Lifecycle Test Goals are not Explicit in the Definition
- 19. Declarative Performance Engineering (DPE) 6 “Enabling the performance analyst to declaratively specify what performance-relevant questions need to be answered without being concerned about how they should be answered. ” [Walter et al.] Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, and Samuel Kounev. Asking ”What”?, Automating the ”How”? -TheVision of Declarative Performance Engineering. In Proc. of ICPE 2016. 91–94. [Walter et al.]
- 20. Declarative Performance Engineering (DPE) 6 “Enabling the performance analyst to declaratively specify what performance-relevant questions need to be answered without being concerned about how they should be answered. ” [Walter et al.] Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, and Samuel Kounev. Asking ”What”?, Automating the ”How”? -TheVision of Declarative Performance Engineering. In Proc. of ICPE 2016. 91–94. [Walter et al.] Developers, Testers, Architects, Performance Analyst …
- 21. DPE: Approaches for Performance Testing 7 [Omar et al.] Towards an automated approach to use expert systems in the performance testing of distributed systems. [Westermann] [Scheuner et al.][Omar et al.] “Enabling the performance analyst to declaratively specify what performance-relevant questions need to be answered without being concerned about how they should be answered. ”[Walter et al.] [Westermann] Deriving Goal-oriented Performance Models by Systematic Experimentation [Scheuner et al.] Cloud Work Bench - Infrastructure-as-Code Based Cloud Benchmarking.
- 22. Our Proposal: BenchFlow 8 Declarative Performance Testing DSL
- 23. Our Proposal: BenchFlow 8 Declarative Performance Testing DSL Model-driven Framework
- 24. Our Proposal: BenchFlow 8 Declarative Performance Testing DSL Model-driven Framework https://github.com/benchflow
- 25. BenchFlow: Elements of a Test 9 Test Definition
- 26. BenchFlow: Elements of a Test 9 Test Definition SUT Configuration and Deployment
- 28. 11 configuration: goal: type: configuration observe: exploration: … Goal Definition (Configuration Test)
- 29. Observed Metrics 12 service_A service_B API_1 … goal: type: configuration observe: workload: API_1: - 90thp_response_time services: service_A: - cpu_avg service_B: - ram_avg …
- 30. Exploration Space 13 service_A service_B API_1 … goal: type: configuration exploration: exploration_space: service_A: resources: - memory: range: 1GB... 5GB step: +1GB - cpus: range: 0.1…0.5 - replicas: 1…4 environment: - SIZE_OF_THREADPOOL: range: 5...100 step: +5 …
- 31. 14 Load Function … goal: load_function: users: 1000 ramp_up: 30s steady_state: 10m ramp_down: 30s … Faban
- 32. 15 Quality Gates service_A service_B API_1 … goal: quality_gates: workload: API_1: 90thp_response_time: <= 250ms services: service_A: cpu_avg: <= 70% …
- 33. 16 Termination Criteria … goal: termination_criteria: test: max_time: 3h failed_experiments: 30% …
- 34. 17 Data Collection … data_collection: client_side: faban: interval: 1s server_side: service_A: resource mysql: environment: SETTING_A : example_setting service_B: - logs - resource … service_A service_B API_1
- 35. 18 Workload service_A service_B API_1 … workload: <name_of_the_workload>: driver_type: "http" popularity: 100% inter_operation_timings: "uniform" operations: - … mix: - … … Faban
- 36. 19 System Under Test Deployment service_A service_B API_1 … sut: configuration: target_service: name: service_A endpoint: / sut_ready_log_check: "Started in (.*)" deployment: service_A: server_1 service_B: server_2 …
- 37. 19 System Under Test Deployment service_A service_B API_1 … sut: configuration: target_service: name: service_A endpoint: / sut_ready_log_check: "Started in (.*)" deployment: service_A: server_1 service_B: server_2 … https://bit.ly/2EBNrDu
- 41. Model-driven Framework Experiment Execution ExplorationExecutionAnalysis Test YAML + SUT Deployment Metrics Failures Result Analysis Goal Exploration Experiment Generation Experiment YAML + SUT Deployment Success Execution 21
- 42. Model-driven Framework Experiment Execution ExplorationExecutionAnalysis Test YAML + SUT Deployment Metrics Failures Result Analysis Goal Exploration Experiment Generation Experiment YAML + SUT Deployment Success Execution 21
- 43. Test Execution Lifecycle 22 Terminated Running Handle Experiment Result Determine Exploration Strategy Determine and Execute Experiments: Experiment Life Cycle [can reach goal] [cannot reach goal] user terminated OR [max_time] Goal Reached Completed with Failure experiment results available [experiments remaining] [all experiments executed] Partially Complete [number_of_experiments] Validate Termination Criteria Terminating Check Quality Gates [quality gates pass] [failed quality gates]
- 44. 23 Running Handle Experiment Result Determine Exploration Strategy Determine and Execute Experiments: Experiment Life Cycle experiment results available [experiments remaining] [all experiment Terminating [quality gates pass] [failed qu Test Execution Lifecycle
- 45. 23 Running Handle Experiment Result Determine Exploration Strategy Determine and Execute Experiments: Experiment Life Cycle experiment results available [experiments remaining] [all experiment Validate Termination Criteria Terminating Check Quality Gates [quality gates pass] [failed qu Test Execution Lifecycle
- 46. 23 Handle Experiment Result Determine and Execute Experiments: Experiment Life Cycle [can reach goal] [cannot reach goal experiment results available [all experiment [number_of_experimen Validate Termination Criteria Check Quality Gates [quality gates pass] [failed qu Test Execution Lifecycle
- 47. 23 Termi iment Result d Execute Experiment ycle [can reach goal] [cannot reach goal] Goal Re Completed w experiment results available remaining] [all experiments executed] Partially [number_of_experiments] ate ation ria Check Quality Gates [quality gates pass] [failed quality gates] Test Execution Lifecycle
- 48. 23 Terminated [cannot reach goal] Goal Reached Completed with Failure [all experiments executed] Partially Complete umber_of_experiments] ality Gates [failed quality gates] Test Execution Lifecycle
- 49. 23 Handle Experiment Result Determine Exploration Strategy Determine and Execute Experiments: Experiment Life Cycle [can reach goal] OR [max_ experiment results available [experiments remaining] [all experiments execut Validate Termination Criteria Terminating Check Quality Gates [quality gates pass] [failed quality ga Test Execution Lifecycle
- 50. 23 Terminated ] Goal Reached Completed with Failure ents g] [all experiments executed] Partially Complete [number_of_experiments] Terminating Quality Gates s] [failed quality gates] Test Execution Lifecycle
- 51. configuration: goal: type: configuration observe: exploration: … Extending the Proposed Approach 24
- 52. configuration: goal: type: stability_boundary observe: exploration: … Extending the Proposed Approach 24 LOAD CONFIGURATION STABILITY_BOUNDARY Available Goals:
- 53. Extending the Proposed Approach 25 Declarative Performance Testing DSL
- 54. Terminated Running Handle Experiment Result Determine Exploration Strategy Determine and Execute Experiments: Experiment Life Cycle [can reach goal] [cannot reach goal] user terminated OR [max_time] Goal Reached Completed with Failure experiment results available [experiments remaining] [all experiments executed] Partially Complete [number_of_experiments] Validate Termination Criteria Terminating Check Quality Gates [quality gates pass] [failed quality gates] Extending the Proposed Approach 25 Declarative Performance Testing DSL
- 55. Approach: Summary 26 Explicit Test Goals in the Definition
- 56. Approach: Summary 26 Explicit Test Goals in the Definition Control on the Test Execution Lifecycle
- 57. Approach: Summary 26 Explicit Test Goals in the Definition Control on the Test Execution Lifecycle Terminated Running Handle Experiment Result Determine Exploration Strategy Determine and Execute Experiments: Experiment Life Cycle [can reach goal] [cannot reach goal] user terminated OR [max_time] Goal Reached Completed with Failure experiment results available [experiments remaining] [all experiments executed] Partially Complete [number_of_experiments] Validate Termination Criteria Terminating Check Quality Gates [quality gates pass] [failed quality gates] Visibility on the Test Execution Lifecycle
- 58. Approach: Summary 26 Static Syntactic and Semantic Validation Explicit Test Goals in the Definition Control on the Test Execution Lifecycle Terminated Running Handle Experiment Result Determine Exploration Strategy Determine and Execute Experiments: Experiment Life Cycle [can reach goal] [cannot reach goal] user terminated OR [max_time] Goal Reached Completed with Failure experiment results available [experiments remaining] [all experiments executed] Partially Complete [number_of_experiments] Validate Termination Criteria Terminating Check Quality Gates [quality gates pass] [failed quality gates] Visibility on the Test Execution Lifecycle
- 59. Approach: Summary 26 Extensible for different Test Goals Static Syntactic and Semantic Validation Explicit Test Goals in the Definition Control on the Test Execution Lifecycle Terminated Running Handle Experiment Result Determine Exploration Strategy Determine and Execute Experiments: Experiment Life Cycle [can reach goal] [cannot reach goal] user terminated OR [max_time] Goal Reached Completed with Failure experiment results available [experiments remaining] [all experiments executed] Partially Complete [number_of_experiments] Validate Termination Criteria Terminating Check Quality Gates [quality gates pass] [failed quality gates] Visibility on the Test Execution Lifecycle
- 60. Future Work 27 Apply to real-world use cases: - Apply the approach in real-world contexts
- 61. Future Work 27 Apply to real-world use cases: - Apply the approach in real-world contexts Add more goals: - Regression - Assess different micro services deployment alternatives
- 62. Future Work 27 Apply to real-world use cases: - Apply the approach in real-world contexts Add more goals: - Regression - Assess different micro services deployment alternatives Leverage performance test execution data: - Prioritise performance test execution - Skip execution of tests
- 63. Future Work 27 Apply to real-world use cases: - Apply the approach in real-world contexts Add more goals: - Regression - Assess different micro services deployment alternatives Leverage performance test execution data: - Prioritise performance test execution - Skip execution of tests Integrate with other frameworks: - ContinuITy framework, to leverage production data
- 64. State of Practice 4 Script/UI Based Integrated in CI Pipelines Performance Tests Execution Automation Google Vizier AutoPerf DataMill CloudPerf End-to-end Tests against REST APIs State of Practice Highlights 28 Declarative DSL Declarative Test Definition 10 configuration: goal: load_function: quality_gates: termination_criteria: data_collection: workload: sut: Model-driven Framework BenchFlow Model-driven Framework 20 CI ServerRepo BenchFlow BenchFlow BenchFlow BenchFlow BenchFlow BenchFlow BenchFlow Extending the Approach configuration: goal: type: configuration observe: exploration: … configuration: goal: type: stability_boundary observe: exploration: … Extending the Proposed Approach 24 LOAD CONFIGURATION STABILITY_BOUNDARY Available Goals: