Manuel Buil, SUSE, Dimitrios Markou, Intracom-Telecom During the Danube release, SFC improved a lot its testing. Apart from adding new test cases, we focused on making them better following several good practices like making libraries more modular and reusable, apply python functionality to improve the integration and leveraging the new capabilities that functest was adding. Apart from that, we added a performance report for ODL which was appreciated by that community. One final challenge we needed to tackle was that SFC functionality was changing in regards to the compute host allocation scheme of the instances (topology). This led us to consider coverage of different topologies for our test cases. This extensive coverage was difficult to achieve, as the number of possible combinations grows combinatorially with the number of hosts and instances, exploding the testing time / resource requirements in the CI. Therefore we opted to use a time multiplexing scheme for our testing, where we test a different topology in every CI run (every day) from a predefined (and expandable) set of characteristic combinations. We would like to share all our experiences, learnings and ideas with the community so that we help other projects and get feedback or other ideas they might be doing which could be reusable for us

2. Smart Testing: Catching More Bugs with
Less Code Through Topology Shuffler
Manuel Buil
Dimitris Markou

3. Introducing OPNFV SFC with a simple Use Case: Parental Control
SDN Network
SF1
Client
Classifier
OVS / VPP
SF1
Classifier
"Copied from Brady Johnson's slides <link>"

4. 4
Compute
ovs
Compute
ovs
Client
Server
SF1
SF2
● OPNFV tests use two
computes for the
deployments
● We can distribute the
four VMs among both
● Each variation provides
a different topology
which is important to
test
Different Topologies

5. Different topologies
5
Compute
ovs
Compute
ovs
Client Server
SF1 SF2
tap
connection
classifie
r

6. Compute Compute
Different topologies
6
ovs ovs
Client Server
SF1SF2
vxlan-gpe
tap
connection
classifier

7. Different topologies
7
Compute
ovs
Compute
ovs
Client Server
SF1 SF2
vxlan
tap
connection
classifier

8. Different topologies
8
Compute
ovs
Compute
ovs
Client Server
SF1 SF2
classifier
vxlan
tap
connection
vxlan-gpe

9. Different topologies
9
Compute
ovs
Compute
ovs
Client Server
SF2 SF1
vxlan
tap
connection
vxlan-gpe
classifier

10. Why is important to test
different topologies?

11. BUG 1
11
vxlan
tap
connection
vxlan-gpe
Compute
ovs
Compute
ovs
Client Server
SF1 SF2
classifier
Why is important to test different topologies?

12. Why is important to test different topologies?
12
vxlan
tap
connection
vxlan-gpe
Compute
ovs
Compute
ovs
Client Server
SF2
classifier
SF1
BUG 2

13. 13
vxlan
tap
connection
vxlan-gpe
Compute
ovs
Compute
ovs
Client Server
SF2
classifier
SF1
Why is important to test different topologies?
BUG 3

14. 14
vxlan
tap
connection
vxlan-gpe
Compute
ovs
Compute
ovs
Client Server
SF1 SF2
classifier ?
Why is important to test different topologies?
BUG 4

15. Why testing different topologies is important?
- We observed empirically that bugs appear on specific topologies
- OVS bug: Packets entering through a vxlan tunnel keep the destination port
- ODL bug: Classifier is only deployed in computes with deployed VNFs
- ODL bug: Packet does not know where to go if SF2 not with server
- ODL bug: Only local traffic is classified
- Conclusions:
- It is important to test as many topologies as possible
- We should take control of the placement of VMs. So far the nova scheduler was
doing it
- OBJECTIVE: NEXT SLIDE

16. GOAL

17. How to test multiple
topologies in OPNFV?

18. OPNFV-SFC scenarios
SFC has currently two scenarios:
● non-HA Only one compute⇒
● HA Two computes⇒
The HA scenario is where we should test the different topologies. We thought about doing it using two
alternatives:
Solution 1: Run every combination
Solution 2: Create one scenario per combination

19. SOLUTIONS
Candidate solution 1: Run every combination
#tests = (#topos) x [#current-sfc-testcases] = (#instances ^ #computes) x [3] ->
# tests = 4^2 x 3 = 48 different tests
#time-per-test = (~1h redeploy) + (~1h test run) = ~2h (different CI run bc of ODL messy cleanup (mbuil))
#total-test-time = #tests x #time-per-test = 48 x 2 = 4 days (or 2 days in 2 servers etc)
Grows quadratically with each new test -> unmanageable (You can go infinite)
Candidate solution 2: Create several scenarios
● Limited hardware in CI
● We might end up monopolizing the CI!

20. SOLUTIONS
Solution 1: Run every combination
16 topologies
} 16 x 3 x 2 hours ⇒ 96 hours per CI run
3 testcases
2 hours

21. Solution 2 : Create several scenarios
Scenario 1 :
Scenario 2:
.
.
.
Scenario N:
Disadvantages !!
1. Monopolizing CI
2. Limited hardware resources

22. Our Solution
1. Rule out equivalent topologies:
Client Server
SF1 SF2
Compute 1
=
Client Server
SF1 SF2
Compute 2
e.g.
Outcome: 6 topologies

23. Our Solution
1 CI run/day = 1 Topology/day
Monday = Topology #1
Tuesday = Topology #2
Wednesday = Topology #3
Thursday = Topology #4
Friday = Topology #5
Saturday = Topology #6

24. Our solution
Multiplex topologies in the CI runs
1. Rule out topologies that are equivalent
a. Example: Endpoints in C1 / VNFs in C2 === Endpoints in C2 / VNFs in C1
2. This leads to 6 indicative topos
3. Run a different topo in each CI run (every day)
#total-test-time = Same as before -> O(#tests)

25. Limitations
* We can only have 7 topologies
→ Propose to releng using a autoincrementing CI-RUN ID as an ENV variable in functest container
* ODL cleanup capabilities are limited in some topologies
→ ODL becomes stateful and tests order matter

26. Thank you!

27. Other improvements
Ovs_logger
→ To fetch interesting logs from the computes in case of failure
SSHUtils
→ To be able to gather information from the installer. It was later used by functest to extend their services
Tacker stuff
→ Keystone fix to allow v3 API
Juan’s heat stuff
→ A way to get the nova instance-id from the tacker vnf-id
virsh-sparsify
--> To reduce the size of the image we use to deploy VNFs
Measure ODL performance
→ Measurements of the classifier time to deploy