Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Data for Testing - Heading for Post Process and Analytics

414 views

Published on

Yujun Zhang, ZTE Corporation, Donald Hunter, Cisco, Trevor Cooper, Intel

The testing community created tens of testing projects, hundreds of testing cases, thousands of testing jobs. Huge amount of testing data has been produced. What comes next, then?

The testing community puts in place tools and procedures to declare testcases/projects, normalize and upload results. These tools and procedures have been adopted so we now have lots of data covering lots of scenarios, hardware, installers.

In this presentation, we shall discuss the stakes and challenges of result post processing.

* How analytics can provide valuable inputs to the community, end users or upstream projects.

* How can we produce accurate indicators, reports and graphs, focus on interpreting / consuming test results.

* How can we get the best of breeds of our result mine?

Published in: Software
  • Be the first to comment

  • Be the first to like this

Big Data for Testing - Heading for Post Process and Analytics

  1. 1. Big Data for Testing Heading for post process and analytics
  2. 2. Speakers Yujun Zhang NFV System Engineer from ZTE Corporation. He is current PTL of QTIP in OPNFV, and creator of MitmStack in OpenStack His main interest focuses on performance testing, analysis and tuning Donald Hunter Principal Engineer in the Chief Technology and Architecture Office at Cisco. He leads the MEF OpenLSO Analytics project which uses PNDA.io as a reference implementation for big data analytics in the MEF LSO Framework. Donald's long-term focus has been software architecture leadership for element management systems, diagnostics and network provisioning applications in Cisco's product portfolio.
  3. 3. Content NOW - what does current test data look like FUTURE - what is expected by the community ANALYTICS - introducing PNDA.io, a platform for analytics SAMPLES - what has been done in other domains NEXT - what shall we do in Euphrates
  4. 4. NOW What does current test data look like?
  5. 5. Till 22nd May, 2017 ● ~160k result records ● 30 projects ● 142 cases ● 45 Pods ● 23 Scenarios Test Data Collected OPNFV TestResults site: http://testresults.opnfv.org/test/swagger/spec.html
  6. 6. Data Schema Top level model project : project name case : case name pod : pod name version : platform version (Arno-R1, ...) installer (fuel, ...) build_tag : Jenkins build tag name scenario : the test scenario (previously version) criteria : the global criteria status passed or failed trust_indicator : evaluate the stability of the test case start_date: date time test started stop_date: date time test stopped details Key Points - Common for all records - Customizable schema in details Schema for results: http://testresults.opnfv.org/test/swagger/spec.html#!/APIs/queryTestResults
  7. 7. Typical Func Test Details FuncTest Details - "details": "duration": " 27.79", "success": "100.00", "nb tests": 12 "module": "authenticate " - "details": "duration": " 80.06", "success": "100.00", "nb tests": 11 "module": "glance " Key Points - Success rate as indicator - Breakdown into modules rally sanity results: http://testresults.opnfv.org:80/test/api/v1/results?case=rally_sanity&last=10&project=functest
  8. 8. Typical Perf Test Details StorPerf Details "status": "OK", "agent_count": 4, "metrics": {...}, "timestart": 1479912550.192721, "volume_size": 1, "pod_name": "intel-pod9", "public_network": "ext-net", "duration": 152.46885204315186, "scenario_name": "ceph_warmup", "disk_type": "SSD" Key Points - Test conditions included in details - Breakdown in metrics storperf results: http://testresults.opnfv.org:80/test/api/v1/results?last=10&project=storperf
  9. 9. Typical Perf Test Metrics StorPerf Metrics "ws.queue-depth.8.block-size.16384.read.iops": 0, "ws.queue-depth.8.block-size.16384.write.latency": 18333.634166666667, "ws.queue-depth.8.block-size.16384.duration": 152, "ws.queue-depth.8.block-size.16384.read.latency": 0, "ws.queue-depth.8.block-size.16384.write.iops": 436.33833333333337, "ws.queue-depth.8.block-size.16384.write.throughput": 6979.75, "ws.queue-depth.8.block-size.16384.read.throughput": 0 Key Points: - Flattened dictionary (not nested) - Dict keys concatenated from metric properties
  10. 10. Report data embedded StorPerf Report Data - "rs.queue-depth.2.block-size.16384": "iops": "read": "steady_state": true, "series": [...], "range": 80.7440000000006, "average": 2566.9578000000006, "slope": -7.916618181818701 "write": ... - “wr.queue-depth.2.block-size.2048”: ... Key Points - Metrics grouped in multi level dict - Data broken down into series - Statistics for each metric generated -
  11. 11. Scenario Reporting functest status: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html yardstick status: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html
  12. 12. Testing could be expensive
  13. 13. FUTURE What is expected by the community?
  14. 14. Values expected from the test data Trend over time Comparison of test results between different SUT or condition Traceability from performance indicator to collected metrics and raw data Detection of anomaly Correlation analysis between performance and SUT factors
  15. 15. Share data, develop collaboratively TESTING PIPELINE TEST COLLECT AGGREGATECALCULATE REPORT Collect metrics by parsing the raw data Calculate indicators and statistics from metrics Aggregate data to create a synthesis from different test cases and iterations Produce raw data Push synthesis data for reporting
  16. 16. Introducing PNDA.io A Platform For Analytics
  17. 17. What is PNDA? PNDA brings together a number of open source technologies to provide a simple, scalable open big data analytics Platform for Network Data Analytics Linux Foundation Collaborative Project based on the Apache ecosystem
  18. 18. Why PNDA? There are a bewildering number of big data technologies out there, so how do you decide what to use? We've evaluated and chosen the best tools, based on technical capability and community support. PNDA combines them to streamline the process of developing data processing applications.
  19. 19. • Simple, scalable open data platform • Provides a common set of services for developing analytics applications • Accelerates the process of developing big data analytics applications whilst significantly reducing the TCO • PNDA provides a platform for convergence of network data analytics PNDA Plugins ODL Logstash OpenBPM pmacct Telemetry Real -time DataDistribution File Store Platform Services: Installation, Mgmt, Security, Data Privacy App Packaging and Mgmt Stream Batch Processing SQL Query OLAP Cube Search/ Lucene NoSQL Time Series Data Exploration Metric Visualisation Event Visualisation PNDA Managed App PNDA Managed App Unmanaged App Unmanaged App Query Visualisation and Exploration PNDA Applications PNDA Producer API PNDA Consumer API PNDA
  20. 20. • Horizontally scalable platform for analytics and data processing applications • Support for near-real-time stream processing and in-depth batch analysis on massive datasets • PNDA decouples data aggregation from data analysis • Consuming applications can be either platform apps developed for PNDA or client apps integrated with PNDA • Client apps can use one of several structured query interfaces or consume streams directly. • Leverages best current practise in big data analytics PNDA Plugins ODL Logstash OpenBP M pmacct Telemetr y Real -time DataDistribution File Store Platform Services: Installation, Mgmt, Security, Data Privacy App Packaging and Mgmt Stream Batch Processing SQL Query OLAP Cube Search/ Lucene NoSQ L Time Series Data Exploration Metric Visualisation Event Visualisation PNDA Managed App PNDA Managed App Unmanaged App Unmanaged App Query Visualisation and Exploration PNDA Applications PNDA Producer API PNDA Consumer API PNDA
  21. 21. SAMPLES What has been done in other domains?
  22. 22. Examples from other domains Event analytics to detect recurring failures, malicious behaviour, future reliability trends https://pndablog.wordpress.com/2017/05/25/an-analytics-based-approach-to-service-assurance-part-2-is -analytics-the-answer/ BGP message analytics to identify cause of unstable AS paths over time https://pndablog.wordpress.com/2017/05/25/bgp-security-how-big-data-can-help-detect-attacks/ Analysis of Openstack VM metrics to detect patterns that lead to loss of service http://pnda.io/usecases https://pndablog.wordpress.com/
  23. 23. Operational Intelligence Planning Intelligence Security Intelligence
  24. 24. NEXT What shall we do in Euphrates?
  25. 25. Roadmap in Euphrates Deploy a PNDA instance in OPNFV infrastructure Sink output from upstream test projects into PNDA instance Develop value-add analysis with dashboards to augment what http://testresults.opnfv.org/reporting/index.html already provides Focus on providing “test intelligence” Prepare path to using PNDA analytics in a production OPNFV world
  26. 26. Questions? https://wiki.opnfv.org/display/testing https://wiki.opnfv.org/display/bamboo/

×