Router Analysis On-Site Training Preview

On-Site Training Preview Proof of Concept Test Overview

Deck Overview• This is a sample deck from a two day instructor led course.• The course consists of presentations, labs and in class discussion.• Each course is tailored for the end user and generally only applies to them.• For more information visit the Router Analysis website or email Steve Noble• All materials are © Router Analysis / SONN

Section 1 Part AWhat is a Proof of Concept Test?

What is a Proof of Concept Test?• A PoC is a test plan designed to prove out a design. You normally call a PoC when you have reached a point in the bidding process where you have > 1 vendor who appears to meet your requirements. – A PoC is used to compare N vendors to see that they fit a set of requirements the customer has listed. – A PoC is used to test each vendor separately and compare the results with the RFP information they were given. – A PoC can be used to help lower the price.

What is a Proof of Concept Test?• It is important to do a Proof of Concept test (PoC). Here are some concepts of a good PoC: – Note: PoCs can go by other names, bake off, bench mark test, etc. – A well designed PoC will allow you to choose not only the better vendor, but be more secure in your choices. – The PoC should be set up in a way that every vendor is able to be directly compared.

Section 1 Part BThe Rules of a PoC

Rule #1Always Discuss and Set the Rules of the PoCPoCs require strict guidelines for each vendor tofollow. Why?• To be able to directly compare results.• To keep everything on the level.• To make sure the PoC is run fairly and you get the right information.

Rule #2Keep Good Communication with the Vendors During the Tests• You will need to grade and keep notes about the tests as they progress to be able to directly compare results.• Vendors will need to keep your own notes and compare them with you after each test and at the end of the day.• This helps vendors to develop a document trail of where they passed/failed and if you are owed any updates after the PoC.

Rule #3Empower Yourself in the Ways of Proper Testing• You need to drive at the PoC including system configuration.• Tests should be based on a 3-5 year plan if possible.• Tests should be combined with each other rather then doing stand alone testing.• Vendors should be required to submit one version of code to be used for all testing.

Rule #4Stay Aware During Testing• Take notice of any configuration changes• If there are hardware or software issues, ask for tracking information

Section 1 Part CTest Plans and Practical Issues

Building Test Plans• The test plan should include clear rules on how the testing will be executed, what equipment is to be provided and the code expected to be on the SUTs.• The testing should be done using a standard IMIX (such as the Light Reading IMIX or Agilent’s IMIX*) not one provided by a specific vendor which favors that vendor.• The test cases should include both bandwidth intensive and PPS intensive tests, not just one or the other.• The test plan should consist of combined tests; i.e. tests that build on-top of each other. Based on your current topology.• Utilize general test cases available from test vendors such as the Agilent Journal of Test Methodologies.

Communicating With Vendors• Discuss the test plan – Work with the vendor to understand common testing issues • Combining tests in a reasonable way • Avoiding the pitfalls of loosely defined test cases – Attempt to define what the grading criteria will be• Tell the vendor what the correct amount of equipment is – Base the hardware on the testing equipment that will be available – Make sure to specifically define the types of interfaces I.e. 10GE LAN• Explain the value of the tests you are running – Explain how your tests are based on your real world goals and expectations

Practical Issues• Many vendor representatives do not have experience doing proper system testing – Confirm the type and number of tester interfaces available • If the testing equipment is limited scale tests, involving traffic will not be very useful. – Check that the vendors will be able to provide the number of interfaces necessary, if not try to come up with ideas to work through the issue. – If the testing is being done on-site, ensure that the testing site is prepared to provide power and HVAC for the equipment provided.

Practical Issues Continued• It is important to understand if a test has passed either fully or conditionally – When doing scale tests define solid numbers. E.g. 1k IS-IS L1 Routes – When doing traffic tests work to avoid running interfaces at 100% as issues such as byte- stuffing and overhead can cause issues. – When combining tests define them as including the last test and be sure to run background traffic from the previous tests to confirm a full pass. – When defining an access list, be sure to provide information about the type such as simple or extended (source/dest ip, source/dest ip/port). – Set time limits, number of re-tries allowed for a test and the time allowed between re- tries before a fail is called. – Define importance to the tests so that vendors know what is required and what the minimum pass is.

Section 1 Part DWhen Things Go Wrong … and they will

Why Things Go Wrong• There will always be issues when tests are being done, most issues revolve around improper configuration of either the SUT or the tester. – To troubleshoot follow the following steps: • Confirm that the System Under Test is connected to the correct ports on the tester and other devices. • Confirm that the ports are all up and functional. • Confirm that all necessary features are configured on both the SUT and the tester. • … More information in the full training.

Issues Completing Tests• There are many reasons why a test may not be completed – Time runs out • Sometimes there is not enough time to do everything the you want. By making sure you understand the value of the different tests you can determine whether to continue or just move on. • Sometimes it may take much longer to configure/debug a test than you expect – Hardware/Software issue • Based on the rules of the BMT you will have different options such as allowing a repeat at a later date, within the next x hours, or giving a fail.

Thank You!

Router Analysis On-Site Training Preview

by Steve Noble

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 62

Statistics