This document discusses OPNFV CI and the challenges faced in implementing continuous integration. It summarizes that while OPNFV CI has achieved running release verification for three major releases and increasing test scenarios and labs over time, there were significant challenges due to OPNFV being a midstream project dependent on upstream stability. Other challenges included slow feedback times, diverse infrastructure environments, lack of community awareness of CI/CD, and differing contributor backgrounds. The document outlines work being done to address these, including rethinking the CI/CD workflow and infrastructure and increasing collaboration with upstream projects.

2. OPNFV CI and Challenges
How we solved them – if we solved them at all!
Fatih Degirmenci
Principal Developer, Ericsson

3. Who am I?
• Principal Developer, Ericsson
• Member of the Technical Steering Committee, OPNFV
• Project Team Lead, Releng, OPNFV
• Committer to several Infrastructure Projects, OPNFV
• Active Technical Contributor, OpenStack

4. What is OPNFV?
Open Platform for Network Functions Virtualization
“OPNFV is a carrier-grade, integrated, open source
platform to accelerate the introduction of new NFV products
and services.”

5. OPNFV is a midstream project
Enhancements
Additions

6. Realizing ETSI NFV Architecture

7. To transform telco

8. Service Providers
Telco/Network Vendors
IT Vendors
Linux Distro Vendors
Education/Research
Institutions
Altogether

9. By doing
DEPLOY
DEVS BUILD &
INTEGRATE
TEST USERS

10. We call it Platform CI
Target POD
(SUT)
Jumphost
Initiate
Run for certain
Scenario
Download
ISO
Deploy Functest
Functional Testing
Yardstick
Platform Benchmarking
Pull Down
Functest Image
Pull Down
Yardstick Image
Labs connected 24x7
using JNLP
Google Cloud
Storage
Docker
Hub
Docker
Hub

11. Using the tools

12. On distributed OPNFV Infrastructure

13. Which is, rather complex…
Lab/POD Ericsson Linux Foundation Orange Intel Huawei
Host OS Ubuntu CentOS
Installer / Cloud Execution Apex Compass Fuel Joid
SDN OpenDaylight ONOS OpenContrail
Testing Functional Testing (Functest) Platform Benchmarking (Yardstick)
Addons / Features SFC BGPVPN KVM for NFV FD.IO

14. But we achieved a lot!
What Release1
Arno
Release2
Brahmaputra
Release3
Colorado
Scenarios supported by CI1 2 20+ 37+
No of Pharos labs2 3 12 14
CPU Architecture3 1 1 2
No of nodes utilized by OPNFV Jenkins 13 90+ 130+
Time to run release verification ~30 hours 400+ hours 600+ hours
Total no of OPNFV deployments in CI4 8000+

15. Despite the Challenges
• Being a midstream project
• Availability of stable versions from upstream
• OPNFV Development Workflow
• Slow feedback
• Diverse infrastructure and lack of uniformity
• Diversity is good but we need some homogeneity
• Community CI/CD Awareness and resistance to change
• Contributors with different backgrounds, mindset & culture

16. Being a midstream project
• OPNFV can go as fast as its slowest upstream…
• In order for OPNFV to do CI, upstream must do CD!

17. OPNFV Development Workflow
Project Team OPNFV gerrit OPNFV CI/Test OPNFV Release OpenStack gerrit OpenStack CI/Test OpenStack Release
Requirement
Patch
Test
Release
Downstream
Release
Test
Long development cycle
Downstream will delay to next release
Slow feedback, > 5 months
OPNFV specific issues cannot be
tested/detected in time
Document

18. OPNFV Workflow with 3rd Party CI
Project Team OPNFV gerrit OPNFV CI/Test OPNFV Release OpenStack gerrit OpenStack CI/Test OpenStack Release
Requirement
Patch
Test
Release
Downstream
Release
Test
Test
Fast feedback
Fast development cycle
Fast feedback, < 1 day
OPNFV specific issues can be
tested/detected in time;
Correction in same release
Document
OPNFV Workflow

19. 3rd Party CI
contribute contribute contribute
pull from master
post feedback

20. Infrastructure Diversity
LaaS
LaaS
LaaS

21. Community CI/CD Awareness
40
168
(+320%)
251
(+49%)
272
(+8%)
13
36
(+176%)
75
(+108%)
84
(+12%)
0
50
100
150
200
250
300
Arno ( 2015-06) Brahmaputra (2016-02) Colorado (2016/09) Now (2016/11)
Total Contributors Contributors to CI

22. In the pipeline
• Re-thinking our CI/CD and Infrastructure
• CI and Infrastructure Evolution
• Further increase the community awareness in CI/CD
• “Recruit” more core people
• Close collaboration with upstream projects on CI/CD
• Push them to CD

23. Questions