The document describes an end-to-end performance testing framework that uses InfluxDB and Grafana for real-time monitoring and analytics. It includes a replay framework that eliminates data loss during performance tests by writing transactions to a CSV file when the database is down and replaying from the CSV when the database comes back up. The framework provides benefits like increased efficiency, accuracy, and scalability for performance and load testing.

1. End-To-End Performance Testing Framework
Large Scale Infrastructure Monitoring

2. Today’s Agenda
- About us
- What we do at Performance Engineering
- What are the monitoring tools used
- How do we use influx-Data stack
- Performance Test Result Validation
- Which Replay Framework do we use
- Benefits and Scalability

3. About US
Senior Developer - Performance Engineering, SAP
Experienced in performing Automation Feasibility Analysis, Test Estimation,
Test Strategy Development, Test Execution for different types of web,
angular and mobile based applications
Sound Implementation Experience is HP QTP(UFT), TestComplete, Selenium,
Robot Framework, HP LoadRunner / Jmeter
Proficient knowledge on Java, Python, VB Scripting, Ansible, Docker, Shell
Scripting, Perl Scripting
Subhodeep Ganguly

4. InfluxDB & Grafana
Why influxDB ?
1 . InfluxDB is a time series, metrics, and analytics database.
2. It’s written in Go and has no external dependencies.
3. Very minimal effort required to manage the DB infrastructure
4 . Once you install it there’s nothing else to manage

5. Efficient Real Time ELK Monitoring

6. Making Difference by Improving Performance

7. High Level Architecture

8. 8INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
 Automated Disk Clean-up Tools for all the servers
 Slack Notification Tool, enabling efficient triggering of alerts
 Replay Tool Integration: Eliminating the possibility of any data loss , which will make the Results
and Logs more accurate.
 Resilient and Disruptive Testing
 Overall Performance Infrastructure Health check Alerts running 24/7 autonomously
 Automated Resolver Tool (developed in ansible) to fix any kind of alerts and onboarding new
services to the existing infrastructure with very minimal/ NO efforts.
 Automation of Dynatrace Agent Deployment & Enablement of Dynatrace Monitoring
Additional Infra Monitoring Modules :

9. Overall Resources – Performance Infrastructure

10. 10INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
 Performance Framework
 Business Impact: End to End performance testing with single click
 Supports Perf Testing using single load generator and multiple load generator scenarios
 Supports Real Time Performance Results Visualization
 Enables Performance Qualification for Five Core Stack Products and Microservices
 Enables perf data collation and Aggregation from various sources:
 JMeter JTL files
 Infrastructure : Host level, container level and Nginx metrics, etc
 Logs : App, Jmeter and Nginx logs
 Performance Aggregate , Performance Logs and Generate Metrics based on requirement
This module is used for generating consolidated aggregated performance metrics and reports for a
particular perf test.
 Enabling Performance Rules to certify tests : This module is used for applying some specific
set of performance rules on performance reports to conclude the test as Pass/Fail
Performance Testing Tools

11. 11INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
High Level Diagram of the Framework : Working Model

12. 12INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
 Performance Framework to influxDB(time-series db) Replay Tool
 Implementation of self-defensive logic to avoid database failures during performance test
 Supports real-time Replay of the Perf-Test logs any time during or after the execution.
 100% Elimination of any Data/Transaction loss during Perf-Test Execution
 Angular Visualization Application – Real Time Performance Visualization App
 Visualization of Performance Test Summary Report in an Angular Application with necessary details
 Validation different type of Metrics i.e. JVM, Node Start-up Time, Response Time, Query Statistics, AWR Reports, Exceptions etc
 Link to other monitoring tolls with a single click
 TIG Stack Tools :
 influxDB for collating performance test result and system logs in PerfResults & SystemAndEvents database
 Telegraf Monitoring Agent in all app and DB hosts for capturing System and App Metrics
 Grafana Visualization Tool to monitor real time End-to-End Performance metrics i.e. CPU/Mem/Disk usage, Writes/sec, Thread count etc
 Upgrade to influxDB latest version 1.7.9 to overcome some challenges during perf-test execution
 High Availability of influxDB using influx relay server (TBD)
 ELK Stack Tools : Elasticsearch – Keep Running Logs from different nodes, Performance Logs, Logstash, Kibana for App-Logs Monitoring
 Capturing Keep Running Logs , PerfLogs, Jmeter & Filebeat logs from all App and DB servers
 Visualization in Kibana UI and monitoring any exceptions or error is any of the hosts
 Upgrade to ELK Version 7.8.x and Reindexing of old Indices to mitigate some of the issues related to Performance Metrics generation.
Performance Testing Tools…continued

13. 13INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
 Dynatrace Enablement – Monitoring of Host Health, Disk/CPU Usage, System/DB Metrics, Method
hotspot using OneAgent deployment.
 Performance Infrastructure and Tools Development in Google Cloud Platform
 This framework can be adoptable to any cloud infrastructure(AWS, GCP, Azure) with very less
manual effort.
 Supports Load, Stress, Stability, Volume and API Testing
 Performance testing can be achieved for any products( i.e. Microservices, Web Based etc)
 Fully based on Open Source Model. Licensing cost is very minimal.
Performance Testing Tools…continued

14. 14INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Aggregate Dashboard – Sample View

15. 15INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Grafana Sample Dashboard

16. 16INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
High Level Architecture : Server Performance Monitoring

17. 17INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
 How do we address any kind of database failures, network failures, connectivity failures
between AUTs , disk space failures or any kind of other failures which affects your execution
 Can we recover any transactions due to these failures after your Automated Performance or
Functional Test Execution is over ?
 In this context, suppose your test execution is time & resource costly, then how do you address
these failures ?
 Do you need to execute the same test case again from scratch ?
 But still do you have any guarantee to get the accurate result in case of any such failures
again?
 Can you verify Run Time Results in Dashboard to certify that your test execution is going fine ?
WHY DO WE NEED A REPLAY FRAMEWORK IN PLACE ?

18. 18INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Automated Solution Approach :

19. 19INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Flowchart of the Replay Tool Implementation :
Start
NoDownload Performance-
Framework jar and start
gathering Test Info
Yes
Start writing the
Transaction/Re
cords in CSV
file
Delete the
CSV File
Check if
CSV File
Exists &&
more than 2
days old ??
Check if influx-
DB is down ?
Set boolean
IsDBDown
= False?
No
Write each
Transaction/Record
in databaseInfluxDB Database
Set boolean
IsDBDown
= True? Yes
Check if IsDBDown =
True ?
Set boolean
IsDBDown
= False?
Write each
Transaction/Record
in database from
CSV File
Yes
No
Create CSV File in
the workspace and
give a Unique Name
suffixing TestID
End after 3 Retries
Retry for 3 times
No
END
Iterated/Executed by Multiple Threads
Store downtime
duration in Timer
Start
Stop
Display in
Dashboard
DashboardComputationLogic

20. 20INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
 100% Elimination of any Data/Transaction loss during Automation or Performance Test
Execution using Jmeter and Real-Time Performance Analytics Looks like Grafana
 Reduction in Effort and man-hours for completing execution of a performance/automation test.
 Elimination of duplicate test-execution effort for performance or automation tests, due to
inaccurate performance metrics, automation results and graphs.
 Enhancement of overall test execution efficiency and fix of db failures with minimal efforts.
 High accuracy of Performance Metrics ensuring better compatibility between baseline and
comparison test(in two different builds)
 This tool can be used as a plugin with any time-series of relational database system
with/without(depending on the database type ) minimal modification.
Benefits after this Implementation

21. 21INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Reference
 Grafana Integration
https://www.blazemeter.com/blog/how-to-use-grafana-to-monitor-jmeter-non-gui-results
 Real Time Test Results Visualization:
https://www.vinsguru.com/jmeter-real-time-results-influxdb-grafana/
 Jmeter Performance Monitoring Model:
http://www.testautomationguru.com/jmeter-server-performance-monitoring-with-collectd-influxdb-grafana/
 End- To End Monitoring:
https://neilkillen.com/2019/09/09/capture-monitor-jmeter-metrics-in-real-time-with-influxdb-grafana
 Replay Framework Model :
https://softwaremill.com/monitoring-apache-kafka-with-influxdb-grafana/

22. 22INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Call for Papers is now open!
We’re looking for great speakers – submit
your speaker application today.
November 10 – 11, 2020
North America Virtual
Experience
www.influxdays.com/virtual-experience-2020/

23. 23INTERNAL© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Thank you !