The document provides an overview of a proof of concept test for evaluating multiple vendors. It discusses establishing clear rules and guidelines for the test to allow for direct comparison of results. Key points covered include communicating well with vendors during testing, empowering yourself in testing approaches, building test plans with standard traffic mixes and combined test cases, and preparing for potential issues that may occur during testing. The goal is to set up the proof of concept test properly to reliably evaluate vendors and choose the best solution.

1. On-Site Training Preview
Proof of Concept Test
Overview

2. Deck Overview
• This is a sample deck from a two day
instructor led course.
• The course consists of presentations, labs and
in class discussion.
• Each course is tailored for the end user and
generally only applies to them.
• For more information visit the Router
Analysis website or email Steve Noble
• All materials are © Router Analysis / SONN

3. Section 1 Part A
What is a Proof of Concept Test?

4. What is a Proof of Concept Test?
• A PoC is a test plan designed to prove out a design. You
normally call a PoC when you have reached a point in the
bidding process where you have > 1 vendor who appears to
meet your requirements.
– A PoC is used to compare N vendors to see that they fit a set of
requirements the customer has listed.
– A PoC is used to test each vendor separately and compare the results
with the RFP information they were given.
– A PoC can be used to help lower the price.

5. What is a Proof of Concept Test?
• It is important to do a Proof of Concept test (PoC). Here are
some concepts of a good PoC:
– Note: PoCs can go by other names, bake off, bench mark test, etc.
– A well designed PoC will allow you to choose not only the better
vendor, but be more secure in your choices.
– The PoC should be set up in a way that every vendor is able to be
directly compared.

6. Section 1 Part B
The Rules of a PoC

7. Rule #1
Always Discuss and Set the Rules of the PoC
PoCs require strict guidelines for each vendor to
follow. Why?
• To be able to directly compare results.
• To keep everything on the level.
• To make sure the PoC is run fairly and you get
the right information.

8. Rule #2
Keep Good Communication with the Vendors
During the Tests
• You will need to grade and keep notes about
the tests as they progress to be able to directly
compare results.
• Vendors will need to keep your own notes and
compare them with you after each test and at
the end of the day.
• This helps vendors to develop a document trail
of where they passed/failed and if you are
owed any updates after the PoC.

9. Rule #3
Empower Yourself in the Ways of Proper Testing
• You need to drive at the PoC including system
configuration.
• Tests should be based on a 3-5 year plan if
possible.
• Tests should be combined with each other
rather then doing stand alone testing.
• Vendors should be required to submit one
version of code to be used for all testing.

10. Rule #4
Stay Aware During Testing
• Take notice of any configuration changes
• If there are hardware or software issues, ask
for tracking information

11. Section 1 Part C
Test Plans and Practical Issues

12. Building Test Plans
• The test plan should include clear rules on how the testing will
be executed, what equipment is to be provided and the code
expected to be on the SUTs.
• The testing should be done using a standard IMIX (such as the
Light Reading IMIX or Agilent’s IMIX*) not one provided by a
specific vendor which favors that vendor.
• The test cases should include both bandwidth intensive and
PPS intensive tests, not just one or the other.
• The test plan should consist of combined tests; i.e. tests that
build on-top of each other. Based on your current topology.
• Utilize general test cases available from test vendors such as
the Agilent Journal of Test Methodologies.

13. Communicating With Vendors
• Discuss the test plan
– Work with the vendor to understand common testing issues
• Combining tests in a reasonable way
• Avoiding the pitfalls of loosely defined test cases
– Attempt to define what the grading criteria will be
• Tell the vendor what the correct amount of equipment is
– Base the hardware on the testing equipment that will be available
– Make sure to specifically define the types of interfaces I.e. 10GE LAN
• Explain the value of the tests you are running
– Explain how your tests are based on your real world goals and expectations

14. Practical Issues
• Many vendor representatives do not have experience doing
proper system testing
– Confirm the type and number of tester interfaces available
• If the testing equipment is limited scale tests, involving traffic will not be very
useful.
– Check that the vendors will be able to provide the number of interfaces necessary, if not
try to come up with ideas to work through the issue.
– If the testing is being done on-site, ensure that the testing site is prepared to provide
power and HVAC for the equipment provided.

15. Practical Issues Continued
• It is important to understand if a test has passed either fully
or conditionally
– When doing scale tests define solid numbers. E.g. 1k IS-IS L1 Routes
– When doing traffic tests work to avoid running interfaces at 100% as issues such as byte-
stuffing and overhead can cause issues.
– When combining tests define them as including the last test and be sure to run
background traffic from the previous tests to confirm a full pass.
– When defining an access list, be sure to provide information about the type such as
simple or extended (source/dest ip, source/dest ip/port).
– Set time limits, number of re-tries allowed for a test and the time allowed between re-
tries before a fail is called.
– Define importance to the tests so that vendors know what is required and what the
minimum pass is.

16. Section 1 Part D
When Things Go Wrong
… and they will

17. Why Things Go Wrong
• There will always be issues when tests are being done, most
issues revolve around improper configuration of either the
SUT or the tester.
– To troubleshoot follow the following steps:
• Confirm that the System Under Test is connected to the correct ports on the tester
and other devices.
• Confirm that the ports are all up and functional.
• Confirm that all necessary features are configured on both the SUT and the tester.
• … More information in the full training.

18. Issues Completing Tests
• There are many reasons why a test may not be completed
– Time runs out
• Sometimes there is not enough time to do everything the you want. By making
sure you understand the value of the different tests you can determine whether to
continue or just move on.
• Sometimes it may take much longer to configure/debug a test than you expect
– Hardware/Software issue
• Based on the rules of the BMT you will have different options such as allowing a
repeat at a later date, within the next x hours, or giving a fail.

19. Thank You!