Education & Training 39SecurityMagazine.com / SECURITY / JUNE 2019 Next Generation Firewall Testing Using Open Standards O rganizations are faced with complex decisions when evaluating what products will improve network secu- rity. There are many factors that go into this type of decision of what products will improve the security of a network. Next-generation firewalls are a critical piece of network security, so they need to be carefully evaluated when purchasing. A next-generation firewall defines the latest evolution in firewalls that take traditional firewall function of packet filtering, network and port trans- lations and stateful inspections adding additional filtering, inspecting and pre- vention of network traffic. Performance of a firewall while executing these functions is important in determining which prod- uct should be selected by an organization. How do you compare performance of firewalls? When comparing firewall performance, there are several places that an organiza- tion could look to get the values. They could go to the product vendors and ask for the performance of their products directly and try to compare. One problem arises with this approach: the values that the firewall might provide could potential- ly not be an “apples-to-apples” comparison but an “apples-to-oranges” comparison. For example, products might report a value of number of packets thru an inter- face. One product might count packets by sending packets thru with a low payload. A second product may count packets that are sent with a size 64k payload. The results for these two devices would be very different based on these testing methods. This makes comparisons of results almost impossible when getting values directly from the products. Another option for an organization when attempting to compare firewall performance results would be to run the testing on their own. First, an organiza- tion would need to figure how to bench- mark a firewall. It would be inefficient to create the test cases, so it would be best to go find requirements for benchmarking a firewall. The Benchmarking Methodology Working Group at the Internet Engineering Task Force (IETF) pro- duced an RFC 3511, “Benchmarking Methodology for Firewall Performance” that documents methods for performance testing of a firewall such as HTTP trans- action, transfer and throughput. These are useful for traditional firewalls but don't cover next-generation firewall benchmarking metrics. There aren’t any defined methods for Intrusion Detection or Prevention that a modern firewall needs to have performance bench- marked. Individual organizations would have to create their own test and make sure they cover all the possible areas of performance that might be of interest. This leads to potential holes in the test- ing, since it doesn’t have a wide review as an IETF document gets as it goes thru the process. Additionally, the self- testing ...

1. Education & Training
39SecurityMagazine.com / SECURITY / JUNE 2019
Next Generation Firewall Testing
Using Open Standards
O
rganizations are faced with
complex decisions when
evaluating what products
will improve network secu-
rity. There are many factors
that go into this type of decision of what
products will improve the security of a
network. Next-generation firewalls are
a critical piece of network security, so
they need to be carefully evaluated when
purchasing. A next-generation firewall
defines the latest evolution in firewalls
that take traditional firewall function of
packet filtering, network and port trans-
lations and stateful inspections adding
additional filtering, inspecting and pre-
vention of network traffic. Performance of
a firewall while executing these functions
is important in determining which prod-
uct should be selected by an organization.
How do you compare performance of
firewalls?

2. When comparing firewall performance,
there are several places that an organiza-
tion could look to get the values. They
could go to the product vendors and ask
for the performance of their products
directly and try to compare. One problem
arises with this approach: the values that
the firewall might provide could potential-
ly not be an “apples-to-apples” comparison
but an “apples-to-oranges” comparison.
For example, products might report a
value of number of packets thru an inter-
face. One product might count packets by
sending packets thru with a low payload.
A second product may count packets that
are sent with a size 64k payload. The
results for these two devices would be very
different based on these testing methods.
This makes comparisons of results almost
impossible when getting values directly
from the products.
Another option for an organization
when attempting to compare firewall
performance results would be to run the
testing on their own. First, an organiza-
tion would need to figure how to bench-
mark a firewall. It would be inefficient to
create the test cases, so it would be best to
go find requirements for benchmarking a
firewall.
The Benchmarking Methodology
Working Group at the Internet
Engineering Task Force (IETF) pro-

3. duced an RFC 3511, “Benchmarking
Methodology for Firewall Performance”
that documents methods for performance
testing of a firewall such as HTTP trans-
action, transfer and throughput. These
are useful for traditional firewalls but
don't cover next-generation firewall
benchmarking metrics. There aren’t any
defined methods for Intrusion Detection
or Prevention that a modern firewall
needs to have performance bench-
marked. Individual organizations would
have to create their own test and make
sure they cover all the possible areas of
performance that might be of interest.
This leads to potential holes in the test-
ing, since it doesn’t have a wide review
as an IETF document gets as it goes
thru the process. Additionally, the self-
testing option isn’t the most efficient use
of resources for each IT department to
repeat the same testing for internal use.
Third-party lab testing is a solution
that allows for one lab to run the testing
and give a report to a product’s company.
The company can then distribute the
report to its customers allowing organi-
zations to evaluate results. Using third-
party reports that allow comparisons
minimizes the amount of testing that
needs to be done. These third-parties cre-
ate test cases and run testing on products
from multiple sources producing a report
with the security performance metrics.
Typically, these third-parties are often

4. neutral, which give organizations more
confidence that the results are taken in
fair manner. The one drawback to third
By Timothy Winters
Contributing Writer
039-40 - Edu & Training - Col_2.indd 39 5/24/19 8:29 AM
40 JUNE 2019 / SECURITY / SecurityMagazine.com
party testing is that it is often closed test-
ing which causes problems for both the
product and organization.
Closed testing is when testing meth-
odologies aren’t available to either the
product being tested or the organizations
that need the results. For product vendors,
this leads to a certain amount of surprise
when results from a testing are revealed.
Often, they get different values when
testing internally that don’t match the
results reported by the closed testing done
by a third party. This is a combination of
not being involved with the testing, but
also not being able to see the test meth-
odology that was used for the testing.
Products understand what configurations
get optimized performance based on the
environment and might try engineer the
product to get better results. While this
might be called “stacking the deck,” it’s
still important to get the input from the

5. product on how performance testing is
executed. “Stacking the deck” means that
a product vendor would only allow testing
that will show favorable results. To prevent
“stacking the deck,” it’s important for
organizations to have access to the testing
methodologies. This allows the organiza-
tion to see what is tested and how it’s
tested to ensure it covers the performance
and security needs of their IT depart-
ments. An organization might notice an
improvement when reviewing test meth-
odology for Common Vulnerabilities and
Exposures (CVE) detection. Products are
easily able to detect CVEs when only the
attack is sent thru the box. But what hap-
pens in the more realistic case that the box
is under load when the CVEs are sent?
Does it continue to detect them or does it
just drop the attacks? These are examples
of ways that open testing helps the entire
community when making the hard choices
for improving network security.
NetSecOPEN is a collection of orga-
nizations, products and third-party test
labs that have a mission of working with
industry to create well defined, open and
transparent standards that reflect the secu-
rity needs of the real world. Its first project
is to focus on Open Performance Testing.
Allowing for the products, organizations
and third-party testers to collaborate on
creating test methodologies. These test

6. methodologies are being brought to
the IETF Benchmarking Methodology
Working Group to address the lack of
benchmarking documents for next-gener-
ation firewalls. These types of open test-
ing programs will allow for organizations
to have “apples-to-apples” comparisons.
About the Author
Timothy Winters is a Senior Executive,
Software and IP Networking, at
the University of New Hampshire
InterOperability Laboratory (UNH-IOL). He
works with companies from all over the
world to develop broad-based, flexible
testing strategies to cost effectively meet network
interoperability
requirements for the Internet Protocol version 6 (IPv6),
Software
Defined Networking (SDN), Session Initiation Protocol (SIP),
Routing
and Home Networking.
Education & Training
autistic child who is settling into a new
school. The emotionally intelligent man-
ager, capable of transformational leader-
ship, has a 360-degree understanding of
their employees and knows when to play
soft vs. hardball.
Small Team Expertise
Military are experienced working with

7. small teams, which are generally from
three to eight members. Military examples
of small teams include artillery teams, large
aircraft crews, surveillance teams, sensor
or warfare teams on warships, armored
vehicle crews, infantry assault groups and
others. In the security sector, small teams
include account teams in service of a client,
technology center personnel, administra-
tive teams, etc. Military veterans bring to
the corporate world the skills they have
honed working with small, nimble teams
that are expert at achieving their goals.
Team Players
Teamwork is a vital lesson all military
veterans learn. In the military, you live and
work together, and are taught to support
your team members and efficiently col-
laborate with the people around you. This
is an invaluable skill in the security sector
whether you are seeking an entry level or
management position.
Workplace Diversity
Veterans represent diversity and collab-
orative teamwork in action having served
with people from diverse economic, eth-
nic and geographic backgrounds as well
as race, religion and gender. According
to Pew Research Center statistics released
in 2017, racial and ethnic minority
groups made up 40 percent of Defense

8. Department active-duty military in 2015,
up from 25 percent in 1990. There is
no place for discrimination or intoler-
ance in the military and security sectors.
Both sectors understand the importance of
treating every person fairly and promoting
job performance.
No Military to Civilian
Decoder Needed
Veterans need a “military to civilian
decoder” system to help explain the sig-
nificance of their military skills and how
they translate to the general employment
landscape. The physical security sector,
however, understands the language of
the military and do not generally require
military responsibilities be coded into lan-
guage that non-military can understand.
The physical security sector features a
wide variety of jobs from entry level, middle
management to senior positions. A retired
veteran with a pension may look to the
security sector for part-time or full-time
entry-level work. Other former military,
who are not eligible for retirement benefits,
may secure mid-level appointments with
the goal of climbing the ladder to the high-
est rungs. The flexibility and opportunity
are unparalleled in the security sector.
Veterans generally enter the workforce
with identifiable skills that can be trans-

9. ferred to the physical security world and
are often skilled in technical trends per-
tinent to business and industry. And what
they don’t know, they are eager to learn –
making them receptive and ready hires in
physical security environments that value
ongoing learning and training.
About the Author
Harold E. Underdown is Vice President
of Training and Development at Allied
Universal. Prior to joining the security
sector, Underdown served a distinguished
30-year career with the United States
Navy as Command Master Chief SEAL
Team FOUR / Master Chief Special Operator.
continued from page 38Raising Your Rank
039-40 - Edu & Training - Col_2.indd 40 5/24/19 8:29 AM
Reproduced with permission of copyright owner. Further
reproduction prohibited
without permission.