Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects

213 views

Published on

Jose Lausuch, Ericsson, Nikolas Hermanns, Ericsson

How can we make sure that new code in OPNFV does not break or stop CI?

How can we ensure quick feedback for each patch-set?

With the new way to snapshot a virtual deployment it is now possible to get virtual clouds up and running in about 2 min. In addition, through low amount of disk/cpu consumption and isolation of the networking it is possible to have a very high number of virtual deployments co-existing in the same bare-metal server.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects

  1. 1. Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects Nikolas Hermanns Jose Lausuch
  2. 2. Outline • Challenges of the SDNVPN project • XCI patchset verification • Virtual testing setup • Functest test challenge
  3. 3. Integration Project (e.g. SDNVPN) ODL SDNVPN Fuel Apex • Consume many different Project • Different release dates • Different Bug tracking • Different source code handling • Different programming language • Etc.
  4. 4. Challenge of Integration Projects (e.g. ODL in OPNFV) ODL-Master time ODL-Release OPNFV-Integration •Bug finding ODL-Release Bug1 ODL-Stable Bug2 Bug1 Bug2 ODL-Master x • Bug 1 found and solved • Bug 2 found but not valid for master anymore • Bug 2 was not worth the effort • Bug 3 will not be found until next release ODL is consumed too late! Bug3
  5. 5. Challenge of Integration Projects (e.g. ODL in OPNFV) Developer A Bug 1 time ODL-Release ODL-Release Bug1 Integrator A • Dev A expert in Part of ODL • Int A expert in Fuel but doesn’t know much about ODL code base • Int A needs to find and understand the bug! • Int A need to talk to Dev A and explain the found bug • Might be Dev A does not even remember
  6. 6. Solutions • Solution 1: More test cases in consumed project • Integrator needs to convince and explain the test to Developer • Test case might be too E2Eish to be created in consumed project • Solution 2: XCI patchset verification • Automated integration (Consume latest state) • Fast feedback (reducing from months to < 1 hour) • No interaction from integrator
  7. 7. New Patchset (Commit) Create Virtual Testing Platform Download/Install new artifact Use standard testing framework e.g. Functest Build ODL artifact 10 min 15 min 35 min Unit test Installer Project e.g. Apex Snapshot 5 min Positive -> Merge commit Negative -> Prevent merge Investigate the logs/Find bugs ODL OPNFV Feedback
  8. 8. Bare Metal (48 threads/256 GB RAM/500 GB SSD) vJenkins slave vController vCompute vCompute vCompute Thread pool – 8 vCPU  10 GB Diskspace  40 GB RAM  Own OVS networks vJenkins slave vController vCompute vController vCompute Snapshot Controller vController vController vController Snapshot Compute vCompute vCompute vCompute ~ 100 GB Diskspace x6 • RAM and CPU limiting factor • Very low disk IO cause no installation process • ~2 mins until Openstack/ODL services up • Easy exchange of snapshot
  9. 9. Using the Snapshot vController vController vController Snapshot Controller vController vController vController ~ 100 GB Diskspace >1GB disk consumption per vHost • Using the base concept of qemu-img
  10. 10. CPU pinning • 4 • 3 8 vCPUs ? -> YES! – sharing is the key. • 1 time vJenkins slave vController vCompute CPU-Load Jenkins- Slave running 8 vCPU Start vCmp vCtrl Started vCmp vCtrl Testing started Testing more API Under- provisioned
  11. 11. Cpu pinning • Other vPod do not get effected
  12. 12. Networking • OVS for complete isolation • vPods using same IPs so every vPod has its own networks • Jenkins-slave for Gateway doing NAT vJenkins slave vController vCompute Bare metal host admin storage private public host NAT internet
  13. 13. Bare Metal (48 threads/256 GB RAM/500 GB SSD) vJenkins slave vController vCompute vCompute vCompute Thread pool – 8 vCPU  10 GB Diskspace  40 GB RAM  Own OVS networks vJenkins slave vController vCompute vController vCompute Snapshot Controller vController vController vController Snapshot Compute vCompute vCompute vCompute ~ 100 GB Diskspace x6 • RAM and CPU limiting factor • Very low disk IO cause no installation process • ~2 mins until Openstack/ODL services up • Easy exchange of snapshot
  14. 14. Example of application: Functest FUNCTEST pep8 unit tests pylint docs py35 py27 test new code on OPNFV deployment
  15. 15. Example of application: Functest • Functest bug might break CI • Needs a way to test each change before merging • Fast feedback to the commit owner • Easy to validate framework changes on Functest • Can be tested on a basic nosdn-nofeature scenario
  16. 16. Example of application: Functest Challenges: • Changes on Feature tests  need specific scenario deployed • Increase verification time of each patchset ( + time to build a docker image) • Identify where the change is (framework, test case, …) to trigger an appropriate test case
  17. 17. Thank You Nikolas Hermanns Nikolas.Hermanns@ericsson.com Jose Lausuch jose.lausuch@ericsson.com

×