Brahmaputra included lots of possible combinations and created the concept of scenario. Continuous Integration allows testing of these combinations and produces lots of results. Functest, Yardstick and all the OPNFV test projects are in charge to create, integrate and run testcases. The production of lots of results for very heterogeneous tests is challenging for all the test projects. Lots of parameters are complex to manage (the POD, the installer, the scenario, ..), that is why success criteria are not always trivial to define. This presentation will explain how Functest & Yardstick are trying to address this complexity in test management.

1. June 20–23, 2016 | Berlin, Germany

2. Carrier Grade Testing Integration
J.Lausuch, Ericsson
M.Richomme ,Orange

3. Carrier grade?

4. Carrier grade?

5. Carrier grade?
Not a single definition
The power of the IT cloud meeting Telco constraints
Or
How to 99.999% with 99.999% subsystems

6. ¿Carrier grade?
A question
of
confidence ?
Robust
Manageable
Scalable
Agile
Legal
constraints
Secure Efficient
Monitoring/Supervision
Troubleshooting
Backup/restore
Legacy support
HA
Geographic redundancy
Downtime / SLA
Resources optimization
Intelligent placement
VNFs easy integration
Multi Mano stacks
Soft/hard update
For our customers
For the system
Critical real-time applications
QoS control
Legal Interception
Data retention
Emergency services

7. Carrier grade?

8. Our first answers

9. OPNFV Testing community
Sanity testing
Performance testing
Maintainability testing
Reliability testing
System testing
Black box testing
Regression testing
Integration testing
Security testing
Acceptance testing
Unit testing
Load testing
Compatibility testing Recovery testing
Usability testing
Comparison testing
Stress testingEnd-to-end testing
Scalability testing
Compliance testing
Efficiency testing
Functional testing

10. OPNFV Testing community
Functional testing
Performance testing
Feature testing

11. OPNFV Testing community

12. From Telco issue to OPNFV projects
Fault management
Resource management
Smooth Upgrade
Multisite
HA
Doctor
Promise
Escalator
Multisite
HA

13. Focus on feature projects
Open vSwitch
SFC
BGPVPN
Copper
…

14. Focus on feature projects
Problem
detected
OPNFV Feature
Project created
Dev in Upstream
project 1
Dev in Upstream
project N
Feature
integrated in
OPNFV
scenario(s)
Feature
tested in CI
Feature
integrated in
OPNFV release
Internal dev
Feature tests dev

15. Focus on CI
automation, tooling and software infrastructure
support for OPNFV projects
Why?
What for?
What? software development infrastructure
repeatability and stability control

16. Focus on Infrastructure
Define the min. HW requirements
to deploy and test OPNFV.
Available to the community for
development and automation.
Used to validate OPNFV releases.

17. The challenges

18. Beyond CI, repeatability
Automating tests is not enough
Understanding behavior changes is mandatory to
Save time (not run systematically all the tests)
Adapt the test strategy according to the environment

19. More features, more scenarios….
Problems:
More Features to be tested…
More Scenarios to be tested…
Days are limited to 24h…
Hardware resources for tests may increase but not exponentially
Solutions:
Reduce the number of tests
Select/promote Telco grade scenarios

20. Telco testing means:
repeat the same test again and again.
stress the system over days/weeks.
KPI compliance.
Reliability
Stability
Performant

21. Here comes MANO…
NFVI and VIM Telco grade foundations are still questionable…still lots
to do
MANO game is very open (descriptors, pragmatic versus formal
approach,..)
Lots of Open Source MANO initiatives
• More tests, more combinations, …
• Time needed before possible momentum

22. What about O&M?
• O&M tests not addressed yet in OPNFV
• Several possible options (nagios, centreon, zabix, skydive,…)
• Shall we test the integration of such tools as part of OPNFV

23. Conclusions

24. Thank you

25. Credits
• Earthquake by Irfan Syler, FreeImages.com Content License
• FIREMEN AT WORK II by Ossian Engmark, FreeImages.com
Content License