The number of production CI environments increased from 2 in Arno to 8 with Brahmaputra, each dedicated to a particular installer version (OPNFV flavor and scenario). In future in order to improve overall robustness of the platform production test resources should be independent from installers and scenarios. Furthermore testing between OPNFV deployments will be needed to ensure the promise of data-center interoperability. This presentation deals with mid and long term challenges of the OPNFV testing infrastructure.

1. June 20–23, 2016 | Berlin, Germany

2. Multi-site OPNFV Testing
Challenges
Trevor Cooper (Intel)
Fatih Degirmenci (Ericsson)
Morgan Richomme (Orange)

3. Agenda
Multi-site Testing Today - Trevor
- Pharos Infrastructure
- CI Today
Multi-site Testing Needs - Morgan
- Advanced Test-cases
- Interoperability
Multi-site Testing Plans - Fatih
- Challenges and Requirements
- Future Plans and Vision for Infra and Testing
Summary and Call to Action - Fatih

4. Multi-site Testing Today

5. Multi-site Testing Introduction
• Motivations for Multi-site Testing
• Geographically and technically diverse test environments
• Test resources independent of installers and scenarios promotes robust platform
• Inter-deployment testing for interoperability between data-centers
• Future real-world use-cases and deployment scenarios
• Pharos Infrastructure
• Infra Working Group: Pharos, Octopus, Releng, Genesis
• Infrastructure labs, process, tools and features
• CI / CD / CT
• Environments increasing in number and complexity
• Process to improve platform robustness
• Challenges of a Distributed Infrastructure
• Managing a distributed resources
• Automating testing
• Distributing tests

6. Pharos Infrastructure
• Current priorities:
• Allocation and management of Infra resources for Colorado
• Pharos change process to manage special requirements
• Lab resource booking and usage tool/dashboard
• Pharos compliance verification/validation
• Dealing with today’s multi-site challenges
• Challenges and how we solved them

7. Production Today
CI Yesterday CI Today
POD1
Installer X
Scenario Y
POD2
PODi
Deploy, run tests
POD1
Installer X
Scenario Y
POD2
PODi
Deploy, run tests
Lab 1
Lab 2
Lab 1
Lab 2

8. Production Evolution - Tomorrow
POD1
Installer X
Scenario Y
POD2
PODi
Deploy, run tests
Lab 1
Lab 2
POD1
Installer X’
Scenario Y’
POD2
Lab 3
CI Tomorrow

9. Infra Improvements
• Lab as a Service (LaaS)
• Accessible virtual sandboxes for development and smoke tests
• More homogeneous environments for development work
• Off-load shortage of bare-metal resources
• Efficient use of Lab Resources
• Strive for CD (provide weekly stable versions)
• Provide dynamic use of lab resources (Dev and CI)
• Verify/validate POD setup for automatic configuration
• All labs/pods/installer configurations for automatic POD configuration
• Jenkins / CI improvements - permissions, scheduling, sandboxes, etc.

10. Multisite Testing Needs

11. Test cases
• Advanced test cases
• OPNFV Multisite Project - evolving
• Brahmaputra -> Use case definition and bug fixes
• Colorado -> Single machine/Multiple OpenStack Instances, Functional Testing, Kingbird
• Future -> Go after true Multisite
• Discussion on multi tenant vIMS for D river
• Interoperability testing

12. An advanced test case: vIMS
Today for Functest test: Metaswitch vIMS + Cloudify orchestrator
installed in a single tenant on one OPNFV POD.
Not acceptable in “real life” (SPOF)
Geographical redundancy on current production IMS system
Architecture Cloud ready (Cassandra ring)
Possible evolutions:
1) test in multi-tenants (local redundancy)
2) Test in multi-sites (geographical redundancy)

13. An advanced test case: vIMS
Need of Multi-sites to evaluate
• the connectivity issues (VPN/Flat)
• the consequence on performance
• The accuracy of placement algorithm:
• Performance (DPDK/ODP area)
• Location (legal constraint) …
• Disaster/recovery scenarios/constraints
Similar considerations on vCDN for next releases

14. An interoperability test
• OPNFV is a great place to automate interoperability
testing…
• Several PODs … with several scenarios
• And a centralized CI

15. An interoperability test: bgpvpn
• SDNVPN project deals with the implementation of the
OpenStack bgpvpn API
• Several possible backends
• ODL
• Neutron Bagpipe
• Nuage
• …
• Use OPNFV to perform interoperability testing would be
valuable

16. An interoperability test: bgpvpn
POD1
os-nofeature-bgpvpn-ha
POD2
os-odl_l2-bgpvpn-ha
Deploy, run tests
BGPVN

17. Multi-site Testing Plans

18. Lessons learned
• Distributed infrastructure is not trivial
• Automating tests/ state synchro towards multi-sites
• We learn by doing…first requests on multi-sites needed
raised in Brahmaputra => Multi-site project

19. Next steps
• Apply learnings
• Bringing up, managing and using distributed infrastructure
• Evolve continuously
• No dramatic changes
• E2E View and holistic approach
• Features will come together one day, creating more complete
Multisite solutions

20. Next steps
• Infrastructure
• Provision, manage and use resources cleverly
• Increase the utilization rather than burning CPU uselessly
• Get ready to enter uncharted territories - security and access policies
• CI
• Evaluate CI to see if it can support Multisite use cases
• Provide feedback that fits to purpose rather than bunch of log files
• Continue applying best practices
• Testing
• Test what needs to be tested when it is needed to be tested

21. Summary
• We already are challenged by what we had in the past
and what we currently have
• The complexity will just increase, requiring us as a
community to work together like a community

22. Call to Action