Network Equipment Testing White Paper

White Paper
on
Best Practices in
Network Equipment Testing
© 2007, Jopasana Software & Systems Ltd.

Authors Atul Nene
Aniket Kavathekar
Graphics Gajanan Kamthe

Executive Summary
So much for the right piece of information at right time! The information hungry systems of today are
putting more and more stress on the network bandwidth which is already shared between the network
members. Be it a wired network or a wireless. Be it the high end Blackberry you use for email, the jukebox
you use to download your favorite online music or the network printer in use for the office, each and every
member dents the bandwidth and affects the network performance. The viruses and worms entering the
networks in increasingly innovative methods do no good to help the performance either.

To maintain the network performance above the acceptable threshold, the network elements participating
in the complex enterprise networks should be made more robust, and as far as possible, more intelligent.
This is especially true for routers and switches and all such types of ‘connecting’ devices that form the
network’s glue. The manufacturers of such devices can test the network devices only to an extent
permissible given the constraints of limited in-house resources. Thus the ball finally lands in the court of a
specialized network equipment testing service provider to ensure and verify the functional efficiency of
network equipment.

The role of network elements like routers and switches becomes eminent in maintaining the optimum
bandwidth and securing the information being exchanged. It becomes essential for a smart tester to test
the network element in various environments which are similar to the end user network. Not only this, the
network element developers also need to be made aware of the ‘live scenario’ at customers premises,
where the element needs to be ‘intelligent’ enough to facilitate normal working irrespective of the
peculiarities caused by the network on case to case basis.

This white paper discusses the best practices that can be deployed by network element manufacturers
and testers to ensure impeccable quality and make the products ‘real-world' ready. Testing practices, as
well as planning and documentation practices are addressed. Given that existing network elements will
be upgraded and newer elements will be developed by manufactures on a continuous basis, these best
practices are highly relevant.

Jopasana Software & Systems Ltd.
17, Electronic Co-operative Industrial Estate,
Pune-Satara Road, Pune 411 009,
Maharashtra, India.
Tel nos.: +91(20)2422 8112, +91(20)2422 7879
http://www.jopasana.com
busdev@jopasana.com

© 2007,Jopasana Software & Systems Ltd.

Table of Contents
1. Introduction .........................................................................................................................2
2. Inside an Enterprise Network ..............................................................................................3
3. Constraints ..........................................................................................................................4
4. Best Test Practices .............................................................................................................5
5. Best Planning and Documentation Practices ......................................................................6
6. Benefits of Best Practices ...................................................................................................9
7. Conclusion ........................................................................................................................10

1. Introduction

Picture this. A ship testing service which tells a frigate manufacturer about the elasticity in ship size
required when the ship is supposed to enter the Bay of Bengal, the Sir Creek, the Panama, the Suez and
the Thames. That sort of testing will suggest improvements in design in turn so that Arabs, Asians, South
Americans would be as ‘equitably happy’ as European and Australian customers.

Any design is a waste if the product is not sensitive to customer’s usage preferences. This rule applies to
every tangible product. From ship building to high-end products in networking and communications, a
round nut definitely can’t fix a square hole!

Enterprise Networks
‘Light bulb testing’ mindset is inadequate in SIDEBAR Enterprise networks today are
software testing highly complex and are made up
of every conceivable piece of
Exhaustive software or hardware testing can bring in the ‘ease hardware from Internet
of use’ inputs to the product developers if the testing is not Telephony phones to handheld
restrained by ‘light bulb testing mentality’1. devices. They comprise
computers running varied kinds
Given the complexity of enterprise networks (see SIDEBAR: of operating systems, enterprise
Enterprise Networks), if the test engineer with the same light software and end-user software.
bulb testing’ mindset is given a network element such as a The network elements that
interconnect these devices are
router or a switch that runs specialized software, the product
themselves so complex that one
may work in his lab given the idealized or clinical conditions, can get very fuzzy about where
but may not at customer’s premise. In the networking domain the software takes over from the
a product manufacturer has to see whether his product is hardware. And not to forget, they
‘sensitive and tunable’ enough in each ‘network environment’. are made up of a combination of
wired and wireless technologies.

1
Imagine you are purchasing a light bulb from an electrician and you want to test it. A ‘typical’ shop owner will
remove the bulb from its packing, connect it to socket, show you the light and say ‘I have tested it and it works!’ This
is what we call ‘light bulb testing mentality’ where everything from the socket width, hole-clearance, voltage limit and
maximum allowed current is standardized and what a test engineer has to do is to show that the circuit gets
completed through the bulb and it works.

Network-Equipment-Testing-Whitepaper.Doc 2 of 10


2. Inside an Enterprise Network
While a product developer may
guarantee a performance assurance
in test-labs, real computer networks
are likely to spring unpleasant
surprises. A neophyte router or a
switch is most likely to get
overwhelmed by the hubbub that
goes on inside an enterprise network
environment. The factors are too
many for listing purposes and they
keep piling up. Let’s tour around the
significant ones.

Harmless residents

Take an example of an organization
of two hundred employees for ease
of consideration. Let’s say the
company has established – as is the
norm in any IT enabled organization
striving to become a globally
connected paperless office – an intranet facility which takes care of all the HR, Administration, Visa
procedures and all the back-end procedures one may imagine to run an organization. A further drill-down
brings in an array of email clients, file servers, print sharing systems and test machines.

To the next level of this come hundreds of folders across Intelligent Network Elements
SIDEBAR

all the two hundred users containing thousands of files of This term is still emerging and hence
variety of types like word documents, jpgs, bmps, pdfs, a dictionary definition as such is not
flash files and so on. Let’s not also forget that these yet available.
individual users are likely to use different communication
However the collective consciousness
modes like Skype talk, Google talk, Internet Messengers
across the networking and telecom
(IMs) like Yahoo, MSN and GAIM. domains loosely identifies an
Intelligent Network Element as a
These users are also likely to be participating in business multifunction device with embedded
as well as social networks like Ryze, LinkedIN and Orkut software that is a part of the
to name a few, that will bring in more traffic for the switch- enterprise network and is capable of
like intelligent network element (see SIDEBAR: Intelligent connecting to and interconnecting rest
Network Elements) to handle. All of the network of the participants of the network, be
they voice and data capable
components and residents mentioned above have a
telephone boxes or sophisticated
different expectation from switch in terms of speed of data servers running internet businesses.
transfer and identification of address. The applications like
intranet score the heaviest as they are used both for The generic functions of a network
storage as well as data transfer at different times. gateway or a router or a switch both
unmanaged and manageable are
associated with these super intelligent
devices and they can work with other
devices using various protocols (TCP-
IP and SS7), technologies (Gigabit
Ethernet and Radio Frequency) and
standards.



Network miscreants
Networks would be incomplete if miscreants are missing. They are of all forms, shapes, sizes and nature.
A traveling data packet as well as the whole file system is susceptible to all sorts of held-ups, frisking,
adultery and destruction.

The routers in these cases need to do a policing act after identifying the severity and level of each. A
virus may drop into the network by breaking into one email account, enter the network of the user by
hiding behind downloaded files and show the real colors once the time is right to do the damage. This can
be an active virus or a dormant virus programmed to be activated with date. Similarly there can be
Trojans, malware programs and spy ware.

Then, there are several hues of worms, adware and BOTS entering the network camouflaged with the
identifiable and ‘well-known’ signatures. A router has to be ‘well-informed’ about all sorts of security
threats these miscreants pose for well functioning of the network. It then also needs to stop them from
entering the network.

Timelessly Timely
“Time is an illusion, lunch time doubly so”, writes Douglas Adams, in his hugely popular title, The
Hitchhikers Guide to the Galaxy. Researchers struggling with predictive techniques for prevention of
network intrusions and anomalies have found a new facet of the word “timeless”. And guess what, giving
plausible credibility to the time-space continuum theory is the fact that network miscreants can appear not
just anytime, but also anywhere in the system. Specialist network administrators who think that they
have locked down and secured every single piece of network equipment just need to wait till a
representative from a customer of their organization pays a visit and wants to plug in his laptop to the
network!

3. Constraints
Network Constraints
Considering the network components and their character, each network is unique in terms needs of data
transfer, vulnerability to security threat, alarm threshold limits, the maximum data rate that is apt for a
specific kind of traffic and the volume of data being transferred. A router, while manning the traffic and
disallowing unwanted packets, needs to be sensitive to these constraints and needs to learn on the job.

For example, in networks designed to carry VOIP traffic on higher priority than other data traffic, the
network elements should enforce the bandwidth limit assigned to other data traffic, so as to keep rest of
the unutilized bandwidth available to possible incoming voice calls. Same is true for thin client
environments like banks, where operational effectiveness is determined by the terminals having a
guaranteed minimum bandwidth dedicated to them.

Cost Constraints
The network constraints mentioned above need to be known before attempting to fine tune network
equipment so as to make it robust. The only way to know these network constraints are by introducing the
newly designed router or switch to a real life computer network. In ideal scenario, the equipment producer
will have to create each and every type of network environment to test the universality of his product.

This is not a practical solution from the investment point of view. It is not financially viable to invest in
elaborate network setups just for the sake of testing a network product. That’s where a testing service
provider comes to the rescue. Such a provider adds value by offering its customizable test setup to
multiple product vendors, thereby reducing costs for each vendor.



Tuning a network equipment vis-à-vis the environment
A smart test engineer can complement a product designer by
• Testing the equipment for the performance parameters
• Setting up the security alarm threshold for that network
• Contributing to the intelligence of router by updating ‘run time environment’ information

This ‘intelligence’ primarily consists of the knowledge of assigning appropriate values to various
configuration parameters, based on the kind of network.
Tunable Parameters
While testing a switch or a router in a ‘close to live’ environment, the test engineer can certainly identify
specific functional as well as security related parameters that, when tuned, affect the elements
performance, in turn affecting the network traffic. These parameters can be generally called as tunable
parameters for enhancing the router performance. The features and functionality that they affect are listed
below.

Hardware Security Management Anomaly Detection
Interface Ports MAC spoofing and flooding Configuration Zero-day worm
Maximum allowed ARP spoofing and poisoning Policy protection
throughput VLAN hopping and double Monitoring Traffic Anomalies
Encrypted throughput tagging Reporting Protocol Anomalies
Number of secured DHCP address exhaustion Event correlation Behavior Anomalies
users Switch impersonation
Additional ports Spanning Tree attacks

4. Best Test Practices
We recommend a set of best test practices that, will help network equipment manufacturers produce
robust products. The first two practices are self explanatory and are generally carried out by
manufacturers as part of their fundamental equipment testing.

Test for expected functionality and performance
Test for unexpected functionality and extreme conditions

These best practices sound benign, and elaborate test planning and execution mechanisms are used to
cover this aspect of testing. Nonetheless, bugs are observed in networking products that are out in the
market and hence paying attention to these two practices is called for. These practices are included here
for completeness.

Test with all possible kinds of applications that could utilize the network
This best practice is self explanatory too, though often, it is either missed in entirety or not paid enough
attention to. Most of the times, what is missed is that a truly comprehensive list of applications is not
identified for testing. We think that infrastructure costs and dearth of acutely skilled test engineers cause
this.



The following practices – from our experience – contribute significantly to the product quality and make
the network elements truly enterprise ready.

False Positives

SIDEBAR
Identify volatile parameters and tune the In programs used to filter spam, a
equipment-under-test for every application. false positive is a legitimate
message mistakenly marked as
This best practice needs a much closer look. Lets take an spam. Messages that are
example. In a normally loaded network, 100+ simultaneous determined to be spam may be
connections from a desktop to an internet site would indicate a rejected by a server or client-side
malicious virus or worm, but there may be a specialized spam filter program and returned
to the sender as bounce e-mail.
application that accesses the internet simultaneously via so One problem with many filtering
many connections for its legitimate business functionality. Let’s tools is that if they are configured
take an example of an email sent from a Gmail account which stringently enough to be
contains hundred images as attachment. As a user downloads effective, there's a fairly high
all the images, it would send a flurry of images appearing one chance of getting false positives.
after another on your desktop. The specialized virus detection The risk of accidentally blocking
program embedded into the specialized controlling network an important message has been
element, will take this event as an indication of a malicious enough to deter many companies
virus or a worm. Thus a legitimate use would be mistaken for a from implementing any anti-spam
measures at all.
virus entry. For such an application, the equipment-under-test
has to be tuned to recognize that applications’ network traffic In a different context, a false
pattern and not raise an alarm, else it would not be deemed as positive, also known as a false
behaving correctly. detection or false alarm, occurs
when an antivirus program
detects a known virus string in an
Test for false positives uninfected file. The file, while not
This best practice is the kind of challenge that gets the infected with an actual virus,
toughest test engineers going (see SIDEBAR: False does contain a string of
Positives). A packet level understanding of an application’s characters that matches a string
network traffic alone will enable an engineer to ensure that from an actual virus.
anomalies detected by the equipment under test are not false A false positive can also occur
when a program performs an
positives.
action, which appears to the
antivirus program to be a virus-
like activity.

5. Best Planning and Documentation Practices
We recommend a few best practices for documentation and reporting of testing carried out under various
network scenarios for the equipment under test. We present these in the form of tear-offs or snippets from
some actual reports documented in the course of live projects in our test labs.

Typically, for a particular piece of network equipment – which is invariably a combination of hardware and
software components – that is being tested, a set of applications that will form the runtime environment is
identified. For each such application a detailed test plan – popularly known as the Test Procedure
Document – is prepared. This piece of documentation specifically targets that particular functionality of
the application which will generate traffic on the network. After all, that is what is of interest in our case.

A plan is also prepared which lists down the sequence and timeframes when the test plans will be written
for all application and the projected time span for their execution. This plan is part of the project planning
and execution function and is tracked regularly for the purpose of reporting status.



Prepare a Test Strategy Note for each application

TearOff 1 is from a test plan and notes the application being executed and the details of the network
configuration during the test, as well as the configuration parameters of the equipment under test. It also
lists down target functionality to be executed and has space for noting observations and details pointing
to where the packet capture data and screenshot capture data – if any – are located.

Track the execution status of individual applications

TearOff 2 is from a
planning document and
helps to track the status
of execution of individual
applications.



Keep special track of applications that cause anomalies in the equipment under test

TearOff 3 denotes a summarized track of the applications that caused anomalies to be seen in the
equipment under test.

It also allows to note whether execution of a particular application is suspended for the current test cycle.
That is where the “Build Used” column will come in handy. That column notes the identifying tag of the
hardware and software combination of the equipment under test. Our experience shows that testing
efforts can be optimized by stopping execution of the application causing problems for this cycle, fixing
the problem in the equipment under test, and testing the suspended application again only when a new
build with the fix is available. Testing of the rest of the applications should of course continue on the
earlier build to get complete coverage.



Track complete test cycles
per build

TearOff 4 is a classical piece from an
execution management document
that records and tracks a complete
test cycle. Weekly and cumulative
totals for projected vs. real numbers
give a good measure of the
effectiveness of the planning process
and the efficiency of the test
execution team.

6. Benefits of Best Practices
Meticulous execution of the said best practices, especially in partnership to an experienced provider of
test services, will benefit product quality in the following manner.

Product Quality Development Process & Costs
• Sets the default performance benchmarks • Reduces costs by leveraging infrastructure and
• Adds to the network elements intelligence people skills of the partner
towards deep threat containment • Generates ideas for future features of the
• Gives maximum coverage of the applications product
• Makes the equipment-under-test most ready • Contributes towards a short and successful
for live deployment beta test program for the product
• Enhances product tunability
• Helps develop security policies for the end-user
organization



7. Conclusion
In the networking communications domain, a
product developer is always perplexed by the
dynamism and diversity of potential markets. An
extra effort in terms of these best test practices will
certainly bring in the most sought after ‘user
friendliness’. That’s where the best stand apart
from the pool of very goods!

If the best practices mentioned in this document
become part of every networking product
developers toolbox, it will contribute to excellent
product quality and hopefully elicit ‘oohs’ and ‘aahs’
towards customer experience, and thereby
enhance end-user comfort.

Once that benchmark in quality is reached by
industry-wide products, one can plan to participate
in the next phase of product intelligence and quality
which is ‘self learning and tuning’ by products of
their runtime environment.

You are welcome to send comments, suggestions
and questions to :
Atul Nene (atuln@jopasana.com).

About Jopasana
Jopasana is a product engineering and software
solutions company operating mainly in the hi-tech
"real-time" and embedded systems domain, helping
customers with complete product realization
services and application engineering services for
end-user applications.

Jopasana provides state-of-art solutions in the area
of networking and telecommunication. The
knowledge about various networking protocols and
protocol stacks; Internet Engineering Task Force
(IETF) RFCs, Industry standards, technologies and
network topologies are key factors in executing
projects in this area.

Jopasana, in Sanskrit, means to "nurture". At
Jopasana, since our inception, in 1990, we have
always strived to nurture relationships with our
customers, our employees, our business partners,
stake holders, and the communities that we operate
in.


Network Equipment Testing White Paper

More Related Content

What's hot

Similar to Network Equipment Testing White Paper

Network Equipment Testing White Paper