Yujun Zhang, ZTE Corporation, Donald Hunter, Cisco, Trevor Cooper, Intel
The testing community created tens of testing projects, hundreds of testing cases, thousands of testing jobs. Huge amount of testing data has been produced. What comes next, then?
The testing community puts in place tools and procedures to declare testcases/projects, normalize and upload results. These tools and procedures have been adopted so we now have lots of data covering lots of scenarios, hardware, installers.
In this presentation, we shall discuss the stakes and challenges of result post processing.
* How analytics can provide valuable inputs to the community, end users or upstream projects.
* How can we produce accurate indicators, reports and graphs, focus on interpreting / consuming test results.
* How can we get the best of breeds of our result mine?
2. Big Data for Testing
Heading for post process and analytics
3. Speakers
Yujun Zhang
NFV System Engineer from ZTE Corporation.
He is current PTL of QTIP in OPNFV, and creator of
MitmStack in OpenStack
His main interest focuses on performance testing,
analysis and tuning
Donald Hunter
Principal Engineer in the Chief Technology and
Architecture Office at Cisco.
He leads the MEF OpenLSO Analytics project which
uses PNDA.io as a reference implementation for big
data analytics in the MEF LSO Framework.
Donald's long-term focus has been software
architecture leadership for element management
systems, diagnostics and network provisioning
applications in Cisco's product portfolio.
4. Content
NOW - what does current test data look like
FUTURE - what is expected by the community
ANALYTICS - introducing PNDA.io, a platform for analytics
SAMPLES - what has been done in other domains
NEXT - what shall we do in Euphrates
7. Till 22nd May, 2017
● ~160k result records
● 30 projects
● 142 cases
● 45 Pods
● 23 Scenarios
Test Data Collected
OPNFV TestResults site: http://testresults.opnfv.org/test/swagger/spec.html
8. Data Schema
Top level model
project : project name
case : case name
pod : pod name
version : platform version (Arno-R1, ...)
installer (fuel, ...)
build_tag : Jenkins build tag name
scenario : the test scenario (previously version)
criteria : the global criteria status passed or failed
trust_indicator : evaluate the stability of the test case
start_date: date time test started
stop_date: date time test stopped
details
Key Points
- Common for all records
- Customizable schema in
details
Schema for results: http://testresults.opnfv.org/test/swagger/spec.html#!/APIs/queryTestResults
12. Report data embedded
StorPerf Report Data
- "rs.queue-depth.2.block-size.16384":
"iops":
"read":
"steady_state": true,
"series": [...],
"range": 80.7440000000006,
"average": 2566.9578000000006,
"slope": -7.916618181818701
"write":
...
- “wr.queue-depth.2.block-size.2048”:
...
Key Points
- Metrics grouped in multi level dict
- Data broken down into series
- Statistics for each metric generated
-
16. Values expected from the test data
Trend over time
Comparison of test results between different SUT or condition
Traceability from performance indicator to collected metrics and raw data
Detection of anomaly
Correlation analysis between performance and SUT factors
17. Share data, develop collaboratively
TESTING PIPELINE
TEST COLLECT AGGREGATECALCULATE REPORT
Collect metrics by
parsing the raw data
Calculate indicators and
statistics from metrics
Aggregate data to
create a synthesis from
different test cases and
iterations
Produce raw data Push synthesis data
for reporting
19. What is PNDA?
PNDA brings together a number of open source technologies to
provide a simple, scalable open big data analytics Platform for
Network Data Analytics
Linux Foundation Collaborative Project based on the Apache
ecosystem
20. Why PNDA?
There are a bewildering number of big data technologies out there,
so how do you decide what to use?
We've evaluated and chosen the best tools, based on technical
capability and community support.
PNDA combines them to streamline the process of developing data
processing applications.
21. • Simple, scalable open data platform
• Provides a common set of services
for developing analytics applications
• Accelerates the process of
developing big data analytics
applications whilst significantly
reducing the TCO
• PNDA provides a platform for
convergence of network data
analytics
PNDA
Plugins
ODL
Logstash
OpenBPM
pmacct
Telemetry
Real
-time
DataDistribution
File
Store
Platform Services: Installation, Mgmt,
Security, Data Privacy
App Packaging
and Mgmt
Stream
Batch
Processing
SQL
Query
OLAP
Cube
Search/
Lucene
NoSQL Time
Series
Data
Exploration
Metric
Visualisation
Event
Visualisation PNDA
Managed App
PNDA
Managed App
Unmanaged
App
Unmanaged
App
Query
Visualisation
and Exploration
PNDA
Applications
PNDA
Producer API
PNDA
Consumer API
PNDA
22. • Horizontally scalable platform for
analytics and data processing
applications
• Support for near-real-time stream
processing and in-depth batch analysis on
massive datasets
• PNDA decouples data aggregation from
data analysis
• Consuming applications can be either
platform apps developed for PNDA or
client apps integrated with PNDA
• Client apps can use one of several
structured query interfaces or consume
streams directly.
• Leverages best current practise in big
data analytics
PNDA
Plugins
ODL
Logstash
OpenBP
M
pmacct
Telemetr
y
Real
-time
DataDistribution
File
Store
Platform Services: Installation, Mgmt,
Security, Data Privacy
App Packaging
and Mgmt
Stream
Batch
Processing
SQL
Query
OLAP
Cube
Search/
Lucene
NoSQ
L
Time
Series
Data
Exploration
Metric
Visualisation
Event
Visualisation PNDA
Managed App
PNDA
Managed App
Unmanaged
App
Unmanaged
App
Query
Visualisation
and Exploration
PNDA
Applications
PNDA
Producer API
PNDA
Consumer API
PNDA
24. Examples from other domains
Event analytics to detect recurring failures, malicious behaviour, future reliability
trends
https://pndablog.wordpress.com/2017/05/25/an-analytics-based-approach-to-service-assurance-part-2-is
-analytics-the-answer/
BGP message analytics to identify cause of unstable AS paths over time
https://pndablog.wordpress.com/2017/05/25/bgp-security-how-big-data-can-help-detect-attacks/
Analysis of Openstack VM metrics to detect patterns that lead to loss of service
http://pnda.io/usecases
https://pndablog.wordpress.com/
28. Roadmap in Euphrates
Deploy a PNDA instance in OPNFV infrastructure
Sink output from upstream test projects into PNDA instance
Develop value-add analysis with dashboards to augment what
http://testresults.opnfv.org/reporting/index.html already provides
Focus on providing “test intelligence”
Prepare path to using PNDA analytics in a production OPNFV world