The document proposes Env2Vec, a deep learning approach to accelerate virtual network function (VNF) testing. Env2Vec learns a universal resource characterization model to detect deviations between inferred and actual resource usage, flagging anomalies. It was evaluated on over 400,000 data points from 600+ real-world testing environments and 125 builds, detecting defects automatically with 86.2-100% accuracy while reducing false alarms. Env2Vec reuses learned environment embeddings to detect problems in previously unseen environments with better performance than other approaches.

1. Env2Vec: Accelerating VNF
Testing with Deep Learning
Guangyuan Piao, Pat Nicholson, Diego Lugones
Nokia Bell Labs, Dublin, Ireland
The 15th European Conference on Computer Systems, 30/04/2020

2. © 2019 Nokia2
Introduction
Current challenges for network engineers adopting modern DevOps:
• Manage hundreds of distributed testbeds each running the new software build
• Test new virtual network function (VNF) builds in shorter development cycles
• Assure carrier-grade quality
Objective:
• Use deep learning (DL) techniques to accelerate VNF testing
• Automate the diagnosis of software defects and anomalous builds
• Point out testing engineers to problematic metrics
Introduction → Problem → Proposal → Evaluation → Conclusions

3. © 2019 Nokia3
Problem: Environments must be Considered!
Introduction → Problem → Proposal → Evaluation → Conclusions
125 different models
Varying accuracy

4. © 2019 Nokia4
Proposed Approach: Env2Vec Anomaly Detection
1. Learn a universal resource characterization model
2. Execute the model in real time
• to detect deviations between the inferred value (𝒚 𝒑
′
) and the actual observed RU of the VNF
• Flag anomaly if there is a significant deviation (larger than γ x standard deviations)
𝒚 𝒑
′
GRUs
FNNs
Lookup Tables
(Embeddings)
characterization
model with DL
Ruhistory (Resource Usage)
CF (contextual features)
• Workload metrics (WM)
• Performance metrics (PM)
EM (environment metadata)
• Testbed, build type, test cases etc.
Inferred resource
usage (RU) (e.g., CPU)
Introduction → Problem → Proposal → Evaluation → Conclusions

5. © 2019 Nokia5
System Overview
1. Testbed data collection
2. Model training (daily)
3. Prediction pipeline
4. Raising alarms
5. Updating the model
Introduction → Problem → Proposal → Evaluation → Conclusions

6. © 2019 Nokia6
Evaluation: Carrier-grade VNF Testing
Dataset
• 600+ real-world testing environments
• 125 build chains (testbed, build type, SUT, and test case)
• 400,000 timesteps/data points measured at 15 minute intervals
Evaluation metrics
• Number of generated alarms
• True and false alarm rate ( AT and AF )
• System with a higher number of generated alarms and higher AT is better
Introduction → Problem → Proposal → Evaluation → Conclusions

7. © 2019 Nokia7
Evaluation: Automating Anomaly Detection in New Builds
Compared approaches
• HTM-AD: AD based on RU time series only
• Ridge: linear regression trained on each environment
• Ridge(ts): considers RU history as features
• RFNNall: Env2Vec without using embeddings
Observations
• Either the priority is raising (1) more performance
problems automatically or (2) highly accurate alarms,
• Env2Vec provides the best performance with γ = 1or 2
Introduction → Problem → Proposal → Evaluation → Conclusions

8. © 2019 Nokia8
Evaluation: Reuse Embeddings in Unseen Environment
Env2Vec
• Reuses learned embeddings (capturing similarity of environments)
• Construct environment embedding for a new unseen environment
• Can detect performance problems better in unseen test executions
lack of data for each unseen env.
Introduction → Problem → Proposal → Evaluation → Conclusions

9. © 2019 Nokia9
Conclusions
• We proposed Env2Vec, which creates a universal easy-to-maintain model
• Env2Vec automatically detects defects and bugs in new software builds by identifying
performance degradation and informing the engineer
• Can be extrapolated to previously unseen environments
• Achieves accuracy between 86.2%-100%, while reducing the false alarm rate by 20.9%-
38.1% compared to other approaches
Introduction → Problem → Proposal → Evaluation → Conclusions

10. Q & A
Contact information:
Guangyuan Piao: guangyuan.piao@nokia-bell-labs.com
Pat Nicholson: pat.nicholson@nokia-bell-labs.com
Diego Lugones: diego.lugones@nokia-bell-labs.com