Title: Networking Essentials Companion GuideAuthor: Cisco Networking AcademyPUBLISHED BY: Cisco PressPUBLICATION DATE: March 2022PRINT LENGTH:544 pagesChapter 20Troubleshoot Common Network Problems Objectives Upon completion of this chapter, you will be able to answer the following questions: · What are some of the approaches used to troubleshoot networks? · What is the process of detecting physical layer problems? · What network utilities can you use to troubleshoot networks? · How do you troubleshoot a wireless network problem? · What are some common Internet connectivity problems? · What outside sources and Internet resources are available for troubleshooting? Key Terms There are no key terms in this chapter. Introduction (20.0.1) Congratulations! You‛ve made it to the final chapter in Networking Essentials. While learning about networking in this course, you‛ve likely developed skills you didn‛t even know you needed. Was the purpose of this whole course learning how to set up a simple network? Of course, network setup is important, but it‛s just part of your learning. What do you do when your network doesn‛t work? This is your real test as a network administrator—to be able to figure out what is wrong, fix it, (and here‛s the tricky part) and not accidentally create a whole new problem. Get good at troubleshooting, and you will always be in demand. The Troubleshooting Process (20.1) Troubleshooting is the process of identifying, locating, and correcting problems. Experienced individuals often rely on instinct to troubleshoot. However, you can use structured techniques to determine the most probable cause and solution to problems.Network Troubleshooting Overview (20.1.1) When troubleshooting, you must maintain proper documentation. This documentation should include as much information as possible about the following: · The problem encountered · Steps taken to determine the cause of the problem · Steps to correct the problem and ensure that it will not reoccur You must document all steps taken in troubleshooting, even the ones that did not solve the issue. This documentation becomes a valuable reference should the same or a similar problem occur again. Even in a small home network, good documentation saves hours of trying to remember how a problem was fixed in the past.Gather Information (20.1.2) When a problem is first discovered in the network, it is important to verify the problem and determine how much of the network is affected by it. After the problem is confirmed, the first step in troubleshooting is to gather information. The following checklist provides some of the important information you should check: · Nature of Problem · End-user reports · Problem verification report · Equipment · Manufacturer · Make/model · Firmware version · Operating system version · Ownership/warranty information · Configuration and Topology · Physical and logical topology · Configuration files · Log files · Previous Troubleshooting · Steps taken · Res ...

1. Title: Networking Essentials Companion GuideAuthor: Cisco
Networking AcademyPUBLISHED BY: Cisco
PressPUBLICATION DATE: March 2022PRINT LENGTH:544
pagesChapter 20Troubleshoot Common Network Problems
Objectives
Upon completion of this chapter, you will be able to answer the
following questions:
· What are some of the approaches used to troubleshoot
networks?
· What is the process of detecting physical layer problems?
· What network utilities can you use to troubleshoot networks?
· How do you troubleshoot a wireless network problem?
· What are some common Internet connectivity problems?
· What outside sources and Internet resources are available for
troubleshooting?
Key Terms
There are no key terms in this chapter.
Introduction (20.0.1)
Congratulations! You‛ve made it to the final chapter in
Networking Essentials. While learning about networking in this
course, you‛ve likely developed skills you didn‛t even know you
needed. Was the purpose of this whole course learning how to
set up a simple network? Of course, network setup is important,
but it‛s just part of your learning.
What do you do when your network doesn‛t work? This is your
real test as a network administrator—to be able to figure out
what is wrong, fix it, (and here‛s the tricky part) and not
accidentally create a whole new problem. Get good at
troubleshooting, and you will always be in demand.
The Troubleshooting Process (20.1)
Troubleshooting is the process of identifying, locating, and

2. correcting problems. Experienced individuals often rely on
instinct to troubleshoot. However, you can use structured
techniques to determine the most probable cause and solution to
problems.Network Troubleshooting Overview (20.1.1)
When troubleshooting, you must maintain proper
documentation. This documentation should include as much
information as possible about the following:
· The problem encountered
· Steps taken to determine the cause of the problem
· Steps to correct the problem and ensure that it will not reoccur
You must document all steps taken in troubleshooting, even the
ones that did not solve the issue. This documentation becomes a
valuable reference should the same or a similar problem occur
again. Even in a small home network, good documentation saves
hours of trying to remember how a problem was fixed in the
past.Gather Information (20.1.2)
When a problem is first discovered in the network, it is
important to verify the problem and determine how much of the
network is affected by it. After the problem is confirmed, the
first step in troubleshooting is to gather information. The
following checklist provides some of the important information
you should check:
· Nature of Problem
· End-user reports
· Problem verification report
· Equipment
· Manufacturer
· Make/model
· Firmware version
· Operating system version
· Ownership/warranty information
· Configuration and Topology
· Physical and logical topology
· Configuration files
· Log files
· Previous Troubleshooting

3. · Steps taken
· Results achieved
One of the first ways to gather information is to question the
individual who reported the problem, as well as any other
affected users. Questions can include end-user experiences,
observed symptoms, error messages, and information about
recent configuration changes to devices or applications.
Next, collect information about any equipment that may be
affected. This information can be gathered from documentation.
A copy of all log files and a listing of any recent changes made
to equipment configurations are also necessary. Log files are
generated by the equipment itself and can usually be obtained
through the management software. Other information on the
equipment includes the manufacturer, make, and model of
devices affected, as well as ownership and warranty
information. The version of any firmware or software on the
device is also important because there may be compatibility
problems with particular hardware platforms.
Information about the network can also be gathered using
network monitoring tools. These tools are complex applications
often used on large networks to continually gather information
about the state of the network and network devices. They may
not be available for smaller networks.
After all necessary information is gathered, you start the
troubleshooting process.Structured Troubleshooting Methods
(20.1.3)
Several structured troubleshooting approaches can be used.
Which one you use depends on the situation. Each approach has
its advantages and disadvantages. Here, you‛ll learn methods
and guidelines for choosing the best method for a specific
situation.Bottom-Up
In bottom-up troubleshooting, you start with the physical layer
and the physical components of the network, as shown in Figure
20-1, and move up through the layers of the OSI model until the
cause of the problem is identified.

4. Figure 20-1 Bottom-Up Troubleshooting
Bottom-up troubleshooting is a good approach to use when the
problem is suspected to be a physical one. Most networking
problems reside at the lower levels, so implementing the
bottom-up approach is often effective.
The disadvantage with the bottom-up troubleshooting approach
is that it requires you to check every device and interface on the
network until the possible cause of the problem is found.
Remember that each conclusion and possibility must be
documented, so a lot of paperwork can be associated with this
approach. A further challenge is to determine which devices to
start examining first.Top-Down
As shown in Figure 20-2, top-down troubleshooting starts with
the end-user applications and moves down through the layers of
the OSI model until the cause of the problem has been
identified.
Figure 20-2 Top-Down Troubleshooting
End-user applications of an end system are tested before
tackling the more specific networking pieces. You can use this
approach for simpler problems or when you think the problem is
with a piece of software.
The disadvantage with the top-down approach is it requires you
to check every network application until the possible cause of
the problem is found. Each conclusion and possibility must be
documented. The challenge is to determine which application to
start examining first.Divide-and-Conquer
Figure 20-3 shows the divide-and-conquer approach to
troubleshooting a networking problem. As network
administrator, you select a layer and test in both directions from
that layer.
Figure 20-3 Divide-and-Conquer Troubleshooting
In divide-and-conquer troubleshooting, you start by collecting
user experiences of the problem, document the symptoms, and
then using that information, make an informed guess as to

5. which OSI layer to start your investigation. When a layer is
verified to be functioning properly, it can be assumed that the
layers below it are functioning. You can work up the OSI
layers. If an OSI layer is not functioning properly, you can work
down the OSI layer model.
For example, if users cannot access the web server but can ping
the server, the problem is above Layer 3. If pinging the server is
unsuccessful, the problem is likely at a lower OSI layer.Follow -
the-Path
Follow-the-path is one of the most basic troubleshooting
techniques. The approach first discovers the traffic path all the
way from source to destination. The scope of troubleshooting is
reduced to just the links and devices that are in the forwarding
path. The objective is to eliminate the links and devices that are
irrelevant to the troubleshooting task at hand. This approach
usually complements one of the other approaches.Substitution
The substitution approach is also called swap-the-component
because you physically swap the problematic device with a
known, working one. If the problem is fixed, the problem is
with the removed device. If the problem remains, the cause may
be elsewhere.
In specific situations, this can be an ideal method for quick
problem resolution, such as with a critical single point of
failure. For example, a border router goes down. In this case,
simply replacing the device and restoring service may be more
beneficial than troubleshooting the issue.
If the problem lies within multiple devices, you might not be
able to correctly isolate the problem.Comparison
The comparison approach, also called the spot-the-differences
approach, attempts to resolve a problem by changing the
nonoperational elements to be consistent with the working ones.
You compare configurations, software versions, hardware, or
other device properties, links, or processes between working
and nonworking situations and spot significant differences
between them.
The weakness of this method is that it might lead to a working

6. solution without clearly revealing the root cause of the
problem.Educated Guess
The educated guess approach is also called the shoot-from-the-
hip troubleshooting approach. This less-structured
troubleshooting method uses an educated guess based on the
symptoms of the problem. Success of this method varies based
on your troubleshooting experience and ability. Seasoned
technicians are more successful because they can rely on their
extensive knowledge and experience to decisively isolate and
solve network issues. With less-experienced network
administrators, this troubleshooting method might be too
random to be effective.Guidelines for Selecting a
Troubleshooting Method (20.1.4)
To quickly resolve network problems, take the time to select the
most effective network troubleshooting method. Figure 20-4
illustrates the methods that could be used when certain types of
problem are discovered.
Figure 20-4 Selecting a Troubleshooting Method
For instance, software problems are often solved using a top-
down approach, whereas hardware-based problems are solved
using the bottom-up approach. New problems may be solved by
an experienced technician using the divide-and-conquer method.
Otherwise, the bottom-up approach may be used.
Troubleshooting is a skill that you develop by doing it. Every
network problem you identify and solve adds to your skill set.
Physical Layer Problems (20.2)
A large proportion of networking problems are related to
physical components or problems with the physical layer.
Physical problems are concerned mainly with the hardware
aspects of computers and networking devices, and the cables
that interconnect them. Physical problems do not include the
logical (software) configuration of devices.Common Layer 1
Problems (20.2.1)
Remember, the physical layer (Layer 1) deals with the physical

7. connectivity of the network devices. Some of the more common
Layer 1 problems include the following:
· Device power turned off
· Device power unplugged
· Loose network cable connection
· Incorrect cable type
· Faulty network cable
· Faulty wireless access point (AP)
To troubleshoot at Layer 1, you first check that all devices have
the proper power supplied and that the devices are turned on.
This solution might seem to be obvious, but many times the
person reporting the problem may overlook a device that is
within the network path from source to destination. Then you
ensure that no errors show on any LEDs that display the
connectivity status. If you‛re on site, visually inspect all
network cabling and reconnect cables to ensure a proper
connection. If the problem is with wireless, verify that the
wireless access point is operational and that wireless settings
are configured correctly.The Sense of Sight
Vision is used to detect problems such as improperly connected
or poorly constructed cables:
· Cables that are not connected
· Cables connected to the wrong port
· Loose cable connections
· Damaged cables and connectors
· Use of the wrong type of cable
Vision also enables you to view the condition and function of
various network devices with LEDs.The Senses of Smell and
Taste
Smell can alert you to components that are overheating when
you‛re troubleshooting. The smell of burning insulation or
components is distinct and is a sure sign that something is
seriously wrong.
The sense of taste is directly related to the sense of smell
because both use the same receptors. You may also taste the
acridness of something burning.The Sense of Touch

8. When troubleshooting, you can use touch to feel for overheated
components as well as to detect mechanical problems with
devices such as cooling fans. These devices usually create a
small vibration in the component that can be detected using
touch. The absence of this vibration or the presence of
excessive amounts of vibration can indicate that the cooling fan
has failed or is about to do so.The Sense of Hearing
Hearing is used to detect major problems such as electrical
issues and the proper operation of cooling fans and disk drives.
All devices have characteristic sounds, and any change from the
normal sounds usually indicates a problem of some
sort.Wireless Router LEDs (20.2.2)
Regardless of whether the fault is present on a wireless or wired
network, one of the first steps in a bottom-up strategy of
troubleshooting should be to examine the LEDs, which indicate
the current state or activity of a piece of equipment or
connection. LEDs may change color or flash to convey
information. The exact configuration and meaning of LEDs
varies between manufacturers and devices. Figure 20-5 shows a
typical wireless router with LEDs indicating power, system,
WLAN, wired ports, and Internet (labeled WAN), USB, and
Quick Security Setup (QSS, also known as Wi-Fi Protected
Setup [WPS]).
Figure 20-5 LED Lights on a Wireless Router
Note
WPS (or QSS) has known vulnerabilities that allow a threat
actor to gain access to your network. Therefore, a security best
practice is to disable this feature. Refer to documentation to
learn how to disable WPS or QSS.
On some devices, a single LED may convey multiple pieces of
information, depending on the current status of the device. It is
important to check the equipment documentation for the exact
meaning of all indicators, but some commonality does exist.
Most devices have activity LEDs, which are often called link
lights. A normal condition is for these LEDs to flash, indicating

9. that traffic is flowing through the port. A solid green light
typically indicates that a device is plugged into the port, but no
traffic is flowing. No light typically indicates one or more of
the following:
· Nothing is plugged into the port.
· There is an issue with the wired or wireless connection.
· A device or port has failed.
· There is a cabling issue.
· The wireless router is improperly configured; for example, a
port was administratively shut down.
· The wireless router has a hardware fault.
· The device does not have power.
Whether the network is wired or wireless, you should verify that
the device and ports are up and functional before spending large
amounts of time trying to troubleshoot other issues.Cabling
Problems (20.2.3)
If the wired client is unable to connect to the wireless router,
one of the first things to check is the physical connectivity and
cabling. Cabling is the central nervous system of wired
networks and one of the most common issues when experiencing
inactivity.
There are several issues to watch for when cabling:
· Be sure to use the correct type of cable. Two types of UTP
cables are commonly encountered in networking: straight-
through cables and crossover cables. Using the wrong type of
cable may prevent connectivity.
· Improper cable termination is one of the main problems
encountered in networks. To avoid this problem, you should
terminate cables according to standards. Terminate all cables on
the same network via the T568A or the T568B termination
standard, not both. Avoid untwisting too much of the wire pairs
during termination. Crimp connectors on the cable jacket to
provide strain relief.
· Maximum cable run lengths exist based on characteristics of
the different cables. Exceeding these run lengths can have a
serious negative impact on network performance.

10. · If connectivity is a problem, verify that the correct ports are
being used between the networking devices.
· Protect cables and connectors from physical damage. Support
cables to prevent strain on connectors and run cable through
areas that will not be in the way.
Troubleshooting Commands (20.3)
A number of software utility programs can help identify
network problems.Overview of Troubleshooting Commands
(20.3.1)
Most of these utilities are provided by the operating system as
CLI commands. The syntax for the commands may vary between
operating systems.
Some of the available utilities include
· ipconfig—Displays IP configuration information
· ping—Tests connections to other IP hosts
· netstat—Displays network connections
· tracert—Displays the route taken to the destination
· nslookup—Directly queries the name server for information on
a destination domainThe ipconfig Command (20.3.2)
When a device does not get an IP address or has an incorrect IP
configuration, it cannot communicate on the network or access
the Internet. On Windows devices, you can view the IP
configuration information with the ipconfig command at the
command prompt. The ipconfig command has several helpful
options, including /all, /release, and /renew.
The ipconfig command, shown in Example 20-1, is used to
display the current IP configuration information for a host.
Issuing this command from the command prompt displays the
basic configuration information, including IP address, subnet
mask, and default gateway.
Click here to view code image
Example 20-1 The ipconfig CommandC:> ipconfigWindows IP
ConfigurationEthernet adapter Ethernet: Media State . . . . . . .
. . . . : Media disconnected Connection-specific DNS Suffix .
:Wireless LAN adapter Wi-Fi: Connection-specific DNS Suffix

11. . : lan Link-local IPv6 Address . . . . . :
fe80::a1cc:4239:d3ab:2675%6 IPv4 Address. . . . . . . . . . . :
10.10.10.130 Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.10.10.1C:>
The ipconfig /all command, shown in Example 20-2, displays
additional information, including the MAC address, IP
addresses of the default gateway, and the DNS servers. It also
indicates whether DHCP is enabled, the DHCP server address,
and lease information.
Click here to view code image
Example 20-2 The ipconfig /all CommandC: >
ipconfig/allWindows IP Configuration Host Name . . . . . . . . .
. . . : your-a9270112e3 Primary Dns Suffix . . . . . . . : Node
Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . :
No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search
List. . . . . . : lanEthernet adapter Ethernet: Media State . . . . .
. . . . . . : Media disconnected Connection-specific DNS Suffix
. : Description . . . . . . . . . . . : Realtek PCIe GBE Family
Controller Physical Address. . . . . . . . . : 00-16-D4-02-5A-EC
DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled
. . . . : YesWireless LAN adapter Wi-Fi: Connection-specific
DNS Suffix . : lan Description . . . . . . . . . . . : Intel(R) Dual
Band Wireless-AC 3165 Physical Address. . . . . . . . . : 00-13-
02-47-8C-6A DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes Link-local IPv6
Address . . . . . : fe80::a1cc:4239:d3ab:2675%6(Preferred)
IPv4 Address. . . . . . . . . . . : 10.10.10.130(Preferred) Subnet
Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . .
. . : Wednesday, September 2, 2020 10:03:43 PM Lease
Expires . . . . . . . . . . : Friday, September 11, 2020 10:23:36
AM Default Gateway . . . . . . . . . : 10.10.10.1 DHCP Server .
. . . . . . . . . . : 10.10.10.1 DHCPv6 IAID . . . . . . . . . . . :
98604135 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-1E-
21-A5-84-44-A8-42-FC-0D-6F DNS Servers . . . . . . . . . . . :
10.10.10.1 NetBIOS over Tcpip. . . . . . . . : EnabledC:>
How can this utility assist in the troubleshooting process?

12. Without an appropriate IP configuration, a host cannot
participate in communications on a network. If the host does not
know the location of the DNS servers, it cannot translate names
into IP addresses.
If IP addressing information is assigned dynamically, the
ipconfig /release command, shown in Example 20-3, releases
the current DHCP bindings. The ipconfig /renew command
requests fresh configuration information from the DHCP server.
A host may contain faulty or outdated IP configuration
information, and a simple renewal of this information is all that
is required to regain connectivity.
Click here to view code image
Example 20-3 The ipconfig/release and ipconfig/renew
CommandsC:> ipconfig/releaseWindows IP ConfigurationNo
operation can be performed on Ethernet while it has its media
disconnected.Ethernet adapter Ethernet: Media State . . . . . . .
. . . . : Media disconnected Connection-specific DNS Suffix .
:Wireless LAN adapter Wi-Fi: Connection-specific DNS Suffix
. : Link-local IPv6 Address . . . . . :
fe80::a1cc:4239:d3ab:2675%6 Default Gateway . . . . . . . . .
:C:> ipconfig/renewWindows IP ConfigurationNo operation can
be performed on Ethernet while it has its media
disconnected.Ethernet adapter Ethernet: Media State . . . . . . .
. . . . : Media disconnected Connection-specific DNS Suffix .
:Wireless LAN adapter Wi-Fi: Connection-specific DNS Suffix
. : lan Link-local IPv6 Address . . . . . :
fe80::a1cc:4239:d3ab:2675%6 IPv4 Address. . . . . . . . . . . :
10.10.10.130 Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.10.10.1C:>
If, after releasing the IP configuration, the host is unable to
obtain fresh information from the DHCP server, it could be that
there is no network connectivity. In this case, you verify that
the NIC has an illuminated link light, indicating that it has a
physical connection to the network. If this does not solve the
problem, the DHCP server or network connections to the DHCP
server may be the issue.

13. Packet Tracer—Use the ipconfig Command (20.3.3)
In this activity, you will use the ipconfig command to examine
IP configuration information on a host.The ping Command
(20.3.4)
Probably the most commonly used network utility is ping. Most
IP-enabled devices support some form of the ping command to
test whether network devices can be reached through the IP
network.
If the IP configuration appears to be correctly configured on the
local host, next, you can test network connectivity by using
ping. The ping command can be followed by either an IP
address or the name of a destination host. In Example 20-4, the
user pings the default gateway at 10.10.10.1 and then pings
www.cisco.com.
Click here to view code image
Example 20-4 The ping CommandC:> ping 10.10.10.1Pinging
10.10.10.1 with 32 bytes of data:Reply from 10.10.10.1:
bytes=32 time=1ms TTL=64Reply from 10.10.10.1: bytes=32
time=1ms TTL=64Reply from 10.10.10.1: bytes=32 time=1ms
TTL=64Reply from 10.10.10.1: bytes=32 time=1ms
TTL=64Ping statistics for 10.10.10.1: Packets: Sent = 4,
Received = 4, Lost = 0 (0% loss),Approximate round trip times
in milli-seconds: Minimum = 1ms, Maximum = 1ms, Average
= 1msC:> ping www.cisco.comPinging
e2867.dsca.akamaiedge.net [104.112.72.241] with 32 bytes of
data:Reply from 104.112.72.241: bytes=32 time=25ms
TTL=53Reply from 104.112.72.241: bytes=32 time=25ms
TTL=53Reply from 104.112.72.241: bytes=32 time=27ms
TTL=53Reply from 104.112.72.241: bytes=32 time=24ms
TTL=53Ping statistics for 104.112.72.241: Packets: Sent = 4,
Received = 4, Lost = 0 (0% loss),Approximate round trip times
in milli-seconds: Minimum = 24ms, Maximum = 27ms,
Average = 25msC:>
When a ping is sent to an IP address, a packet known as an echo
request is sent across the network to the IP address specified. If

14. the destination host receives the echo request, it responds with a
packet known as an echo reply. If the source receives the echo
reply, connectivity is verified by the reply from the specific IP
address. The ping is not successful if a message such as request
timed out or general failure appears.
If a ping command is sent to a name, such as www.cisco.com, a
packet is first sent to a DNS server to resolve the name to an IP
address. After the IP address is obtained, the echo request is
forwarded to the IP address and the process proceeds. If a ping
to the IP address succeeds but a ping to the name does not, there
is most likely a problem with DNS.Ping Results (20.3.5)
If ping commands to both the name and IP address are
successful but the user is still unable to access the application,
the problem most likely resides in the application on the
destination host. For example, the requested service might not
be running.
If neither ping is successful, network connectivity along the
path to the destination is most likely the problem. If this occurs,
it is common practice to ping the default gateway. If the ping to
the default gateway is successful, the problem is not local. If
the ping to the default gateway fails, the problem resides on the
local network.
In some cases, the ping may fail, but network connectivity is
not the problem. A ping may fail due to the firewall on the
sending or receiving device, or a router along the path that is
blocking the pings.
The basic ping command usually issues four echoes and waits
for the replies to each one. It can, however, be modified to
increase its usefulness. The options listed in Example 20-5
display additional features available.
Click here to view code image
Example 20-5 Options for the ping CommandC: > pingUsage:
ping [-t] [-a] [-n count] [-l size] [-f] [-i TTL] [-v TOS]
[-r count] [-s count] [[-j host-list] | [-k host-list]] [-w
timeout] [-R] [-S srcaddr] [-c compartment] [-p] [-4] [-
6] target_nameOptions: -t Ping the specified host

15. until stopped. To see statistics and continue -
type Control-Break; To stop - type Control-C.
-a Resolve addresses to hostnames. -n
count Number of echo requests to send. -l size
Send buffer size. -f Set Don't Fragment flag in
packet (IPv4-only). -i TTL Time To Live. -v
TOS Type Of Service (IPv4-only. This setting has been
deprecated and has no effect on the type of
service field in the IP Header). -r
count Record route for count hops (IPv4-only). -s
count Timestamp for count hops (IPv4-only). -j host-list
Loose source route along host-list (IPv4-only). -k host-list
Strict source route along host-list (IPv4-only). -w timeout
Timeout in milliseconds to wait for each reply. -R
Use routing header to test reverse route also (IPv6-only).
Per RFC 5095 the use of this routing header has been
deprecated. Some systems may drop echo requests if
this header is used. -S srcaddr Source address to use.
-c compartment Routing compartment identifier. -p
Ping a Hyper-V Network Virtualization provider address. -4
Force using IPv4. -6 Force using IPv6.C:>
Packet Tracer—Use the ping Command (20.3.6)
In this activity, you will use the ping command to examine end-
to-end connectivity between hosts.Divide and Conquer with
ping (20.3.7)
Connectivity problems occur on wireless networks, wired
networks, and networks that use both. When you‛re
troubleshooting a network with both wired and wireless
connections, it is often best to troubleshoot using a divide-and-
conquer technique to isolate the problem to either the wired or
wireless network. The easiest way to determine if the problem is
with the wired or wireless network is to do the following:
· Ping from a wireless client to the default gateway. This
verifies whether the wireless client is connecting as expected.
· Ping from a wired client to the default gateway. This verifies

16. whether the wired client is connecting as expected.
· Ping from the wireless client to a wired client. This verifies
whether the wireless router is functioning as expected.
After the problem is isolated, it can be corrected.The tracert
Command (20.3.8)
Although ping is the most commonly used network
troubleshooting command, other useful commands are available
on Windows devices.
The ping command can verify end-to-end connectivity.
However, if a problem exists and the device cannot ping the
destination, the ping command does not indicate where the
connection was really dropped. To find this dropped connection,
you must use another command known as traceroute or tracert.
Microsoft Windows uses the tracert command, whereas other
operating systems commonly use the traceroute command.
The tracert utility provides connectivity information about the
path a packet takes to reach the destination and about every
router (hop) along the way. It also indicates how long a packet
takes to get from the source to each hop and back (round-trip
time). The tracert utility can help identify where a packet may
have been lost or delayed due to bottlenecks or slowdowns in
the network.
In Example 20-6, the user traces the path to Cisco. The path is
unique to this user. Your path will have a different listing of
hops and may be shorter or longer (number of hops).
Click here to view code image
Example 20-6 Tracing a Route to CiscoC:> tracert
www.cisco.comTracing route to e2867.dsca.someispedge.net
[104.95.63.78]over a maximum of 30 hops: 1 1 ms 1 ms
<1 ms 10.10.10.1 2 * * * Request timed out. 3
8 ms 8 ms 8 ms 24-155-250-94.dyn.yourisp.net
[172.30.250.94] 4 22 ms 23 ms 23 ms 24-155-121-
218.static.yourisp.net [172.30.121.218] 5 23 ms 24 ms
25 ms dls-b22-link.anotherisp.net [64.0.70.170] 6 25 ms
24 ms 25 ms dls-b23-link.anotherisp.net [192.168.137.106] 7
24 ms 23 ms 21 ms someisp-ic-341035-dls-

17. b1.c.anotherisp.net [192.168.169.47] 8 25 ms 24 ms 23
ms ae3.databank-dfw5.netarch.someisp.com [10.250.230.195]
9 25 ms 24 ms 24 ms a104-95-63-
78.deploy.static.someisptechnologies. com
[104.95.63.78]Trace complete.C:>
Note
In the output of Example 20-6, notice that the second hop
failed. The reason is most likely due to a firewall configuration
on that device that does not permit responding packets from the
tracert command. However, the device does forward the packets
to the next hop.
The basic tracert command allows only up to 30 hops between a
source and destination device before it assumes that the
destination is unreachable. This number can be adjusted by
using the -h parameter. Other modifiers, displayed as options in
Example 20-7, are also available.
Click here to view code image
Example 20-7 The Options for the tracert CommandC: >
tracertUsage: tracert [-d] [-h maximum_hops] [-j host-list] [-w
timeout] [-R] [-S srcaddr] [-4] [-6]
target_nameOptions: -d Do not resolve addresses
to hostnames. -h maximum_hops Maximum number of hops
to search for target. -j host-list Loose source route along
host-list (IPv4-only). -w timeout Wait timeout
milliseconds for each reply. -R Trace round-trip
path (IPv6-only). -S srcaddr Source address to use
(IPv6-only). -4 Force using IPv4. -6
Force using IPv6.C:>The netstat Command (20.3.9)
Sometimes you need to know which active TCP connections are
open and running on a networked host. The netstat command is
an important network utility that you can use to verify those
connections. As shown in Example 20-8, the netstat command
lists the protocol in use, the local address and port number, the
foreign address and port number, and the state of the
connection.
Click here to view code image

18. Example 20-8 The netstat CommandC:> netstatActive
Connections Proto Local Address Foreign Address
State TCP 10.10.10.130:58520 dfw28s01-in-f14:https
ESTABLISHED TCP 10.10.10.130:58522 dfw25s25-in-
f14:https ESTABLISHED TCP 10.10.10.130:58523
dfw25s25-in-f14:https ESTABLISHED TCP
10.10.10.130:58525 ec2-3-13-132-189:https ESTABLISHED
TCP 10.10.10.130:58579 203.104.160.12:https
ESTABLISHED TCP 10.10.10.130:58580
104.16.249.249:https ESTABLISHED TCP
10.10.10.130:58624 52.242.211.89:https ESTABLISHED
TCP 10.10.10.130:58628 24-155-92-110:https
ESTABLISHED TCP 10.10.10.130:58651 ec2-18-211-133-
65:https ESTABLISHED TCP 10.10.10.130:58686 do-
33:https ESTABLISHED TCP 10.10.10.130:58720
172.253.119.189:https ESTABLISHED TCP
10.10.10.130:58751 ec2-35-170-0-145:https ESTABLISHED
TCP 10.10.10.130:58753 ec2-44-224-80-214:https
ESTABLISHED TCP 10.10.10.130:58755 a23-65-237-
228:https ESTABLISHEDC:>
Unexplained TCP connections can pose a major security threat
because they can indicate that something or someone is
connected to the local host. Additionally, unnecessary TCP
connections can consume valuable system resources, thus
slowing down the performance of the host. netstat should be
used to examine the open connections on a host when
performance appears to be compromised.
Many useful options are available for the netstat command. You
can view these options by typing netstat /? at the command
prompt, as shown in Example 20-9.
Click here to view code image
Example 20-9 Options for the netstat CommandC: > netstat
/?Displays protocol statistics and current TCP/IP network
connections.NETSTAT [-a] [-b] [-e] [-f] [-n] [-o] [-p proto] [-r]
[-s] [-t] [-x] [-y] [interval] -a Displays all
connections and listening ports. -b Displays the

19. executable involved in creating each connection or
listening port. In some cases well-known executables host
multiple independent components, and in these cases the
sequence of components involved in creating the connection
or listening port is displayed. In this case the executable
name is in [] at the bottom, on top is the component it called,
and so forth until TCP/IP was reached. Note that this option
can be time-consuming and will fail unless you have sufficient
permissions. -e Displays Ethernet statistics. This may
be combined with the -s option. -f Displays
Fully Qualified Domain Names (FQDN) for foreign
addresses. -n Displays addresses and port numbers in
numerical form. -o Displays the owning process ID
associated with each connection. -p proto Shows
connections for the protocol specified by proto; proto
may be any of: TCP, UDP, TCPv6, or UDPv6. If used with the -
s option to display per-protocol statistics, proto may
be any of: IP, IPv6, ICMP, ICMPv6, TCP, TCPv6,
UDP, or UDPv6. -q Displays all connections, listening
ports, and bound nonlistening TCP ports. Bound
nonlistening ports may or may not be associated with
an active connection. -r Displays the routing table. -s
Displays per-protocol statistics. By default, statistics are
shown for IP, IPv6, ICMP, ICMPv6, TCP, TCPv6, UDP, and
UDPv6; the -p option may be used to specify a subset
of the default. -t Displays the current connection
offload state. -x Displays NetworkDirect connections,
listeners, and shared endpoints. -y Displays
the TCP connection template for all connections.
Cannot be combined with the other options. interval
Redisplays selected statistics, pausing interval seconds
between each display. Press CTRL+C to stop redisplaying
statistics. If omitted, netstat will print the current
configuration information once.C:>The nslookup Command
(20.3.10)
When a network device is being configured, one or more DNS

20. server addresses are provided that the DNS client can use for
name resolution. Usually, the ISP provides the addresses to use
for the DNS servers. When a user application requests to
connect to a remote device by name, the requesting DNS client
queries the name server to resolve the name to a numeric
address.
Computer operating systems also have a utility called nslookup
that enables you to manually query the name servers to resolve
a given host name. This utility can also be used to troubleshoot
name resolution issues and to verify the current status of the
name servers.
In Example 20-10, when the nslookup command is issued, the
default DNS server configured for your host is displayed. The
name of a host or domain can be entered at the nslookup
prompt. The nslookup utility has many options available for
extensive testing and verification of the DNS process.
Click here to view code image
Example 20-10 Looking Up Cisco Information with the
nslookup CommandC:Users> nslookupDefault Server: dns-
sj.cisco.comAddress: 171.70.168.183> www.cisco.comServer:
dns-sj.cisco.comAddress: 171.70.168.183Name: origin-
www.cisco.comAddresses: 2001:420:1101:1::a
173.37.145.84Aliases: www.cisco.com>
cisco.netacad.netServer: dns-sj.cisco.comAddress:
171.70.168.183Name: cisco.netacad.netAddress:
72.163.6.223>
Syntax Checker—The nslookup Command (20.3.11)
Practice entering the nslookup command in both Windows and
Linux.
Refer to the online course to complete this activity.
Lab—Troubleshoot Using Network Utilities (20.3.12)
In this lab, you will complete the following objectives:
· Interpret the output of commonly used network command-line
utilities.

21. · Determine which network utility can provide the necessary
information to perform troubleshooting activities in a bottom-up
troubleshooting strategy.
Troubleshoot Wireless Issues (20.4)
Troubleshooting a wireless LAN is similar to troubleshooting a
wired LAN; however, some important differences are associated
with the wireless signal and access point.Causes of Wireless
Issues (20.4.1)
If a wireless client is unable to connect to an AP, the problem
may be due to wireless connectivity problems. Wireless
communications rely on radio frequency (RF) signals to carry
data. Many factors can affect your ability to connect hosts using
RF:
· Not all wireless standards are compatible. The 802.11ac (5
GHz band) is not compatible with the 802.11b/g/n standards
(2.4 GHz band). Within the 2.4 GHz band, each standard uses
different technology. Unless specifically configured, equipment
that conforms to one standard may not function with equipment
that conforms to another. In Figure 20-6, the 2.4 GHz network
is configured to support legacy devices.
Figure 20-6 Basic Wireless Settings on a Wireless Router
· Each wireless conversation must occur on a separate,
nonoverlapping channel. Some AP devices can be configured to
select the least congested or highest throughput channel.
Although automatic settings work, manual setting of the AP
channel provides greater control and may be necessary in some
environments.
· The strength of an RF signal decreases with distance. If the
signal strength is too low, devices are unable to reliably
associate and move data. The signal may be dropped. The NIC
client utility can be used to display the signal strength and
connection quality.
· RF signals are susceptible to interference from outside
sources, including other devices functioning on the same

22. frequency. A site survey should be used to detect for this
interference.
· APs share the available bandwidth between devices. As more
devices associate with the AP, the bandwidth for each
individual device decreases, causing network performance
problems. The solution is to reduce the number of wireless
clients using each channel.Authentication and Association
Errors (20.4.2)
Modern WLANs incorporate various technologies to help secure
the data on the WLAN. Incorrect configuration of any of these
can prevent communication. Some of the most common settings
that are configured incorrectly include the SSID, authentication,
and encryption.
· The SSID is a case-sensitive, alphanumeric string that is up to
32 characters. It must match on both the AP and client. If the
SSID is broadcast and detected, this is not an issue. If the SSID
is not broadcast, it must be manually entered onto the cl ient. If
the client is configured with the wrong SSID, it will not
associate with the AP. Additionally, if another AP is present
that has broadcasted the SSID, the client may automatically
associate to it.
· On most APs, open authentication is configured by default,
allowing all devices to connect. If a more secure form of
authentication is configured, a key is necessary. Both the client
and AP must be configured with the same key. If the keys do
not match, authentication will fail, and the devices will not
associate.
· Encryption is the process of altering the data so that it is not
usable by anyone without the proper encryption key. If
encryption is enabled, the same encryption key must be
configured on both the AP and the client. If the client associates
with the AP but cannot send or receive data, the encryption key
may be the issue, as shown in Figure 20-7.
Figure 20-7 Failed Authentication

23. Packet Tracer—Troubleshoot a Wireless Connection (20.4.3)
In this activity, you will be given a scenario. You will
determine the reason why a wireless client is unable to connect
to a wireless router and correct the problem.
Common Internet Connectivity Issues (20.5)
Several connectivity issues can be attributed to other devices
such as the DHCP server or with reaching the ISP.DHCP Server
Configuration Errors (20.5.1)
If the physical connection to the wired or wireless host appears
to be connecting as expected but the host cannot communicate
on remote networks or the Internet, you should check the IP
configuration of the client.
The IP configuration can have a major impact on the ability for
a host to connect to the network. A wireless router acts as a
DHCP server for local wired and wireless clients and provides
IP configuration (see Figure 20-8). This includes the IP address,
subnet mask, default gateway, and commonly, the IP addresses
of DNS servers. The DHCP server binds the IP address to a
client MAC address and stores that information in a client table.
It is usually possible to view this table using the configur ation
GUI included with the router.
Figure 20-8 DHCP Configuration Received from the ISP
The client table information should match the local host
information, which you can see using the ipconfig /all
command. Additionally, the IP address on the client must be on
the same network as the LAN interface of the wireless router.
The LAN interface of the wireless router should be set as the
default gateway. If the client configuration information does not
agree with information in the client table, the address should be
released (ipconfig /release) and renewed (ipconfig /renew) to
form a new binding.
In most cases, the wireless router receives its own IP address
through DHCP from the ISP. Check to make sure that the router
has an IP address and, if necessary, attempt to release and

24. renew the address using the GUI utility.Check Internet
Configuration (20.5.2)
If hosts on the wired and wireless local network can connect to
the wireless router and with other hosts on the local network but
not to the Internet, as shown in Example 20-11 and Figure 20-9,
the problem may be in the connection between the router and
the ISP.
Click here to view code image
Example 20-11 A Failed PingC:> ping 10.18.32.12Pinging
10.18.32.12 with 32 bytes of data:Request timed out.Request
timed out.Request timed out.Request timed out.Ping statistics
for 10.18.32.12: Packets: Sent = 4, Received = 0, Lost = 4
(100% loss),
Figure 20-9 Sample Topology
There are many ways to verify connectivity between the router
and the ISP. Using the GUI, one way to check connectivity is to
examine the router status page. As shown in Figure 20-10, it
should show the IP address assigned by the ISP (64.100.0.11 in
this example).
Figure 20-10 ISP Configuration
If this page shows no connection, the wireless router may not be
connected. In that case, check all physical connections and LED
indicators. If the DSL or cable modem is a separate device,
check those connections and indicators as well. If the ISP
requires a login name or password, check that they are
configured to match those given by the ISP. Using the GUI,
password configurations can normally be located on the Setup
configuration page. Next, try to reestablish connectivity by
clicking the Connect, or IP Address Renew, button on the status
page. If the wireless router will still not connect, contact the
ISP to see whether the issue is occurring from that end.Check
Firewall Settings (20.5.3)
If Layers 1 through 3 all appear to be operating normally and
you can successfully ping the IP address of the remote server, it

25. is time to check the higher layers. For example, if a network
firewall is used along the path (see Figure 20-11), it is
important to check that the application TCP or UDP port is open
and no filter lists are blocking traffic to that port.
Figure 20-11 Firewalls in a Small Topology
If all clients are obtaining the correct IP configuration and can
connect to the wireless router but are unable to ping each other
or cannot access a remote server or application, the problem
may be with rules on the router. Check all settings on the router
to ensure no security restrictions could be causing the issue.
Verify that the local firewalls on the client devices are not
preventing network functionality.
Customer Support (20.6)
Knowing where to find help when needed is an important part of
being able to solve networking issues.Sources of Help (20.6.1)
If, during the troubleshooting process, you are unable to
determine the problem and its resolution, you might need to
obtain assistance from outside sources. Some of the most
common sources for help include these:
· Documentation—Good documentation can save you a great
deal of time and effort in finding the most likely cause of the
problem. It can also provide the technical information required
to isolate, verify, and correct the issue. The documentation
provided with many networking devices, however, often does
not provide sufficient information to troubleshoot anything
except the most basic issues.
· Online FAQs (Frequently Asked Questions) —Most
manufacturers provide a series of FAQs about their product or
technology on their website. Usually based on previous requests
for help, FAQs are a good source of current information and
should be consulted whenever possible.
· Internet searches—With the increased availability of support
forums, you can now obtain assistance with troubleshooting
from people around the world in real time.

26. · Colleagues—Colleagues are often a wealth of information;
there is no substitute for troubleshooting experience.When to
Call for Help (20.6.2)
Sometimes you cannot solve networking issues by yourself. In
this case, you might need to contact the vendor or ISP support
desk for assistance, as shown in Figure 20-12. The customer
support line or support desk is the first stop for end-user
assistance. The support desk is a group of individuals with the
knowledge and tools required to help diagnose and correct
common problems. It provides assistance to determine whether
a problem exists, the nature of the problem, and the solution.
Figure 20-12 A Customer Support Call
Many companies and ISPs establish support desks to assist users
with networking problems. Most large IT companies run support
desks for their individual products or technologies. For
example, Cisco Systems offers support desk assistance for
problems integrating Cisco equipment into a network or
problems that may occur after installation.
There are many ways to contact a support desk, including email,
live chat, and phone. Although email is good for nonurgent
problems, phone or live chat is better for network emergencies.
This access is especially important in organizations such as
banks where small amounts of downtime can cost large amounts
of money.
If necessary, the support desk can take control of a local host
through remote-access software. This capability allows support
desk technicians to run diagnostic programs and interact with
the host and network without having to physically travel to a
job site. Remote access greatly reduces the wait time for
problem resolution and allows the support desk to assist more
users.Support Desk Interaction (20.6.3)
When you‛re an end user, it is important to give the support
desk as much information as possible, as shown in Figure 20-13.
The support desk will require information on any service or
support plans that are in place along with specific details of the

27. affected equipment. This information can include make, model,
and serial number, along with the version of firmware or
operating system running on the device. They may also require
the IP and MAC address of the malfunctioning device. The
support desk will require information specific to the problem:
Figure 20-13 Providing Information to Customer Support
· What symptoms were encountered?
· Who encountered the problem?
· When did the problem manifest?
· What steps have been taken to identify the problem?
· What were the results of steps taken?
If this is a follow-up call, you should be prepared to provide the
date and time of the previous call, the ticket number, and name
of the technician. Be located at the affected equipment, and be
prepared to provide the support desk staff with access to the
equipment if requested.Issue Resolution (20.6.4)
A support desk is generally organized in a series of levels of
experience and knowledge. If first-level support desk staff are
unable to solve the problem, they may escalate the problem to a
higher level. Higher-level staff are generally more
knowledgeable and have access to resources and tools that the
first-level support desk staff do not.
Record all information regarding the interaction with the
support desk, such as
· Time/date of call
· Name/ID of technician
· Problem reported
· Course of action taken
· Resolution/escalation
· Next steps (follow-up)
When you work together with the support desk, most problems
can be resolved quickly and easily. When the problems are
resolved, be sure to update all documentation accordingly for
future reference.Support Desk Tickets and Work Orders (20.6.5)
When a Level 1 support desk technician receives a call, there is

28. a process followed to gather information. There are also
specific systems for storing and retrieving relevant information.
It is extremely important to gather the information correctly in
the event that a call has to be escalated to a Level 2 technician
or require an onsite visit.
The information gathering and recording process starts as soon
as the technician answers the phone. After customer
identification, the technician accesses the relevant customer
information. Typically, a database application is used to manage
the customer information.
The information is transferred to a trouble ticket, also known as
an incident report. This document can be a piece of paper in a
paper filing system or an electronic tracking system designed to
follow the troubleshooting process from beginning to end. Each
person who works on the problem is expected to record what
was done on the trouble ticket. When an onsite call is required,
the trouble ticket information can be converted to a work order
that the onsite technician can take to the customer site.
When a problem is resolved, the solution is documented in the
customer work order (see Figure 20-14) or trouble ticket, and in
a knowledge base document for future reference.
E
;iJ(Dv!..-='o>
X - o 91
q; (O f oj;3=xd.+o=,
fDOAA:'='rDO-X-
6-f^vu-(D!l
a,LO'=f,Y-u
;tu.-c-=('..r'",U J U C U i
! N' S r' rd < c i E 7_-qt-=-,u.rYj

29. :="(I<=n*?o<=.;oqtr-))^!-. 1
r:-<-946-7
r. ; = - < - :f
r,D ^,r ) ^ J 'D
->-Ao-?rf,<g;
i3p3:io=rDuul[.+
irDQ-r5O=-:i=:lX -s.3e 3u _ v' y -
^ d
C) :-+OA+!
a6O;,1 tulJ_.tu-
Sdbi 5'x;x -
,</Dc)5='u;'=-U--.=-i-
-"1 rro_CaboJ=':/a
oi,ia:lnh+ tuC
oliJ.oi9:=-iX:omrDoj-
=^,-.f='-.U=.^^i
b-+^.,ta}H&; slj<= :
;-;<OuOuiX:
5f 9fl pPq;9eo-; P:;5 --
5nuio-.Ylq.';>of-/D,D:
fr jx cl3: ,^;s5 =3i3 :43.6oio<:
is'5 $q q3a-.+{uo
==:
:0?6- - ='
::. = =:::-
n
!

30. -.--A'- : g !
9;:7A
-.-F-.(,r::*
5/='/a
-L-;-1
. )- '_.
=aa-^:-
>o-P
>!a
-i v.xi_;.1 .
'. : ,:
:u x.-:_*
1a-r
1_
.:-
i)
n)-t
j'a
a>i
,x-
itr -i 3I..-:
7V:-t '< _
^' : -.::*-:
--1
a
4
/.
/il-

31. F
-
h
.o
n
-Ar;_
ziax
YE
rO-
l>
;o.
in-o
o
m
v
N)
O
:
!
:
-
l
::
;.rpE

32. da'
d;
-oR
n*
fri
>t
DaD3
D:i
lq. l
rtl
fr:
ti.
{:
h-:
oa-
i::
6r
^:
-_
>l
a:
:
9o
i+ gr-9ri:FU :;
N .]-
{s

33. =-dDn'
-E
fir
i>
.(Dia
l.Di
'-iri
iD:,H{
E{
7J5a
Fda
L)<
?>
=ir€*
=-=<:
!,
FU
4o
o
F
d
F
!)r)oo
(Dl
A.,
.Dj
pi
'a
N):

34. N:I
,=
_!
a-
L'
-J.<*-
-*
* !
-/-
:.'
-z I
: tJ
:
.
a
6:
"1
'1:_(!X
l^
N.: l-(Jr-
N-
;
a

35. ;
a
:
a-
=r.
a:
a-
'aa
.
a
(.iJ
Y
l..Js(!
I
N)
Ur
F
o
al
Ud -.
na)
Ein
4,2--<
X .-

36. J
'J1
);:
'fr
),J
N
o
'(N
?
H
|-
o
-
?
oa
A
:_
=_-
:- -cn:
=r.
!(D
,,<) .)
1r
:fr
t?(
;

37. it
=(
X;
-un
a*
-j
^?:t
-l 0a
-=
HJ
91- +E ^'6:
+j
s,!
!n
!= ,)-o={-
r'n
JE
oc i
J- (
)U.)F:
=?;)
->";
=_
z--.::
:
l_-
1-

38. F
Eo
L.
s
t.)
o
N)
NI(l) (
!-
I^)'
tI
N)l(x,
6ol
a
c
-/_ -_
.l
=o'Dj
zd
D-
i'
nl
)O
i+
.o
I*
a

39. J:^
.a
'a
L:,
,:^
s
a
?,
..:
:-
:
=
a) t:
:,
:+
)a
a'riz
Da
)<I r,:
l. n
io
)i
!i
^l
,L

40. -J
=<:-
j-
=;: -'
:-
-=
:o
CY
)o!
o
,o
o
a-
5
/D
t
c,
c'
=(
,t
1!:..
i/
:!
:
.:

41. ..'
t)s
i
NJ
90
9n
,< 5'
'^ 9-
p9
5' jJ
.O
P ^
Q=
'-l -
i/t {
ej
j+.
Ce
-o
.!7
i:
=:
=
o

42. : (l'-o
!':5
JJ
-!:
jti
- t':
p
:
=I
:
:
:
_J
j ii
a/
-=
/,
':z
:

43. -:
'r'
A
lll
s
m
=oc
trt
ll!
=oz
t,
g!.i!.Pn
< --l
O+
CO
6
=
!,U:1 at
<o
rDo
Oo
=1sla
f^

44. gc
O ^'
a<
1lo
aU
>rD
'-N
<:
l-+
U6ac
6!
3E
=o
J.<
t- !.
^Oof
fn
6f)
Cz
^!
:.
fg,
."iH
-4<--l )

45. =<.=.rD6;Pd'v' h 6 :r.n?rOq==.E
i<5n='
s:r(o
A)(]U
(D-Or=
nilr,o
Ol-*aha-T
'-'f(D0
Ito H- ;
=xa'-+
^L-^! *-or10ire2=
<-=21.e.apqfr =6 o.'-o)E,D
=6€q)tlnoJ
of
_.-o ) r
iodP
-i- .<xaD-o)
:T<?
3aep
o <. 6'E
^^tt
=.^rDo
9d!o
='^rL-
-d:u,)UJ
u-='6
='o-nrD=-o
^,oi!=
-al-^d-n8i
9dorn
! J ) tvo)o)o_<
A-!I.
=ACA

46. '!JJoJ 4
o
AI
I
o
J
0,
=CLoa
o
o
E€
o
CL
!
o
?
*&04
p a
=Cclc5!
o p + 5
[email protected]
o
o
Eo
o

47. o.
6
a.
o
<
E
oa
Computing (2019) 101:989–1014
https://doi.org/10.1007/s00607-018-0626-5
Instance launch-time analysis of OpenStack
virtualization technologies with control plane network
errors
Jawad Ahmed1,3 · Aqsa Malik1 ·
Muhammad U. Ilyas1,2 · Jalal S. Alowibdi2
Received: 9 October 2017 / Accepted: 8 May 2018 / Published
online: 28 May 2018
© Springer-Verlag GmbH Austria, part of Springer Nature 2018
Abstract We analyzed the performance of a multi-node
OpenStack cloud amid dif-
ferent types of controlled and self-induced network errors
between controller and
compute-nodes on the control plane network. These errors
included limited band-
width, delays and packet losses of varying severity. This study
compares the effects of

48. network errors on spawning times of batches of instances
created using three different
virtualization technologies supported by OpenStack, i.e.,
Docker containers, Linux
containers and KVM virtual machines. We identified
minimum/maximum thresh-
olds for bandwidth, delay and packet-loss rates below/beyond
which instances fail
to launch. To the authors’ best knowledge, this is the first
comparative measurement
study of its kind on OpenStack. The results will be of particular
interest to designers
and administrators of distributed OpenStack deployments.
Keywords OpenStack · Containers · Performance measurement
B Muhammad U. Ilyas
[email protected]; [email protected]; [email protected]
Jawad Ahmed
[email protected]; [email protected]
Aqsa Malik
[email protected]
Jalal S. Alowibdi
[email protected]
1 Department of Electrical Engineering, School of Electrical
Engineering and Computer Science
(SEECS), National University of Sciences and Technology
(NUST), Islamabad 44000, Pakistan
2 Department of Computer Science, Faculty of Computing and
Information Technology,
University of Jeddah, Jeddah, Saudi Arabia
3 School of Electrical Engineering and Telecommunications,
University of New South Wales,
Sydney NSW 2052, Australia

49. 123
http://crossmark.crossref.org/dialog/?doi=10.1007/s00607-018-
0626-5&domain=pdf
http://orcid.org/0000-0003-0308-4361
990 J. Ahmed et al.
Mathematics Subject Classification 68M01 · 68M20 · 68M15
1 Introduction
1.1 Motivation and problem statement
Cloud computing is a paradigm in which pools of physical
resources are shared by
multiple users using virtualization technologies. The shared
resources include storage,
compute, bandwidth, software, development platforms, services
etc. and are available
to users on-demand. OpenStack is an open source cloud
management system (CMS)
that provides infrastructure-as-a-service, i.e., it is used to
launch and control virtual
machine (VM) instances deployed on shared physical servers.
The Telecommunication
industry is looking at virtualizing network functions on Cloud
environments. However,
the layers of software abstractions needed to virtualize
resources creates an overhead
and reduces performance. Another impact factor is isolation
between VNFs co-located
onthesamemachine.VM-
basedvirtualizationrequiresahypervisorthatloadsseparate

50. operating systems for each VM. In contrast, Linux containers
(LXC) do not require
loading another OS, but use kernel namespaces and provide an
alternative to Linux
VMs with a much smaller overhead in terms of CPU and
memory resources. Linux
Container is an attractive technology compared with virtual
machines given its low
resource footprint.
Physical resources in cloud data centers are usually
oversubscribed by a certain
factor, i.e., users arecollectivelyallocatedmoreresources
thanarephysicallyavailable.
This means that on occasion it is possible that resource
utilization demand on a server
may exceed physical capacity. In this situation, practically
imperfect tenant isolation
may lead a VM’s operation to be affected by the actions of
other tenants, which
is in contravention to the principle of perfect tenant isolation in
cloud data centers.
Imperfect tenant isolation on the CPU, memory, disk and
network plane has also been
the subject of study in the context of covert channel
communication between two
co-located VMs without the explicit exchange of messages, e.g.,
Wu et al. [38], Xu
et al. [39] and Ristenpart et al. [31]. While covert channels are
not the subject of this
study, we are studying the effects of different network
conditions on the launch time
of VMs, LXCs and Docker containers. These network conditions
may be caused by
spikes in demand or errors in data center networks.

51. Another motivation for this study stems from the fact that an
increasing number
of clouds are now deployed as distributed clouds. Inter -data
center communication
between geographically separated data centers is at the mercy of
the Internet, where
link conditions may see greater variation.
The objective of this project is to quantify and compare the
effect of network errors
of varying severity levels on the resilience of virtualization
technologies (KVM, LXC
and Docker containers) in OpenStack.
123
Instance launch-time analysis of OpenStack virtualization… 991
1.2 Limitations of prior art
Every day more cloud data centers are being deployed globally.
Cloud service
providers, such as Amazon, HP, Rackspace, etc., offer different
cloud service mod-
els to customers. However, they provide little in the way of
service guarantees for
launch times of VMs/containers in the face network
connectivity issues. Most cloud
performance analysis studies are straightforward benchmark
studies such as the ones
on Amazon EC2 measuring CPU performance and disk I/O
[1,25]. Garfinkel [9] mea-
sured service performance of a number of Amazon software-as-
a-service offerings.

52. 1.3 Proposed approach
For this study we deployed a cloud using the OpenStack cloud
management sys-
tem (CMS) and developed a re-usable benchmarking tool,
developed in Python. The
benchmarking tool measures the resilience of an OpenStack
deployment by induc-
ing delays, packet losses and bandwidth bottlenecks of various
levels into the control
plane network. OpenStack deployments using different instance
virtualization tech-
nologies, i.e., hypervisors, Linux containers and Docker
containers, are benchmarked
by the time it takes to launch batches of instances of different
sizes.1 The functional
and architectural differences of these three virtulization
technologies will be explained
later.
1.4 Experimental results and findings
WedeployedOpenStackonfourphysicalservers,i.e.,onecontrollera
ndthreecompute
nodes. We present the results of three different virtualization
technologies supported
by OpenStack. The results show that the Linux containers in
nova-lxd offer rapid
deploymentandoutperformothervirtualizationtechnologiesduetot
heirlightresource
overhead. The results show that the spawning times increase
linearly at first and then
increase at a greater rate when network errors exceed a certain
threshold.

53. 1.5 Key contributions
The principal contributions of this research work are two-fold:
– We provide first measurements of spawning times in the face
of network turbulence
for three OpenStack virtualization technologies.
– This is among the first comparative measurement studies to
include all kinds of
network faults covering three different virtualization
technologies.
1 OpenStack documentation uses the term instance to refer to a
VM. We will use the terms VM and instance
interchangeably.
123
992 J. Ahmed et al.
2 Related work
2.1 Software defined networking
Computer networks consist of a complex mix of devices, i.e.,
routers, switches,
firewalls etc. These devices can only be configured using their
own vendor propri-
etary software. The proprietary and vendor controlled nature of
the software severely
restricts customer freedom and in turn limits the degree to
which they can innovate.
Software defined networking (SDN) has revolutionized the way

54. networks and new
services are designed, operated and managed [7]. SDNs are
fundamentally different
from traditional networks in that they decouple their control and
data planes. A central
SDN controller is used to control the data plane which is
distributed across switches
and routers of a network. Several different SDN controllers
have been developed over
the last several years, e.g., Beacon [6], Floodlight [5], NOX
[11], ONIX [18], etc.
2.2 Network virtualization
SDN is an enabling technology for network virtualization. SDN
network virtualization
is an abstraction that decouples the physical from the logical
data plane. Network vir-
tualization enables quick (re)configuration of logical networks
[24]. Mininet [13,19]
is an emulator for networks of OpenFlow switches, hosts and
controllers. A single
physical server running Mininet can instantiate hundreds of
hosts with as many con-
trollers on a virtual network. In legacy networks, virtualization
of network devices
such as the firewalls and routers is difficult, but SDN has made
it possible to create
virtual instances of these devices as needed.
2.3 Cloud computing
Cloud computing uses virtualization for compute, storage and
networking resources
to create pools of resources and makes them available to users
on-demand. Network

55. virtualization in particular has given cloud service providers the
flexibility to deploy
complex network configurations orders of magnitudes faster.
Several vendors offering
cloud services include Amazon EC2, Microsoft Azure, Google
Cloud, etc. Sefraoui
et al. [34] conducted a comparative study of different IaaS
providers and how these
cloud service providers have better resource utilization than
legacy technologies.
Network-as-a-Service (NaaS) [4] is a relatively newer cloud
service model. This
service can help users create network topologies and even use
custom routing proto-
cols, and emulate network devices such as switches and routers.
SDN technologies
have had a major impact on cloud computing. The ability of
SDN technologies to allow
vendors to (automate) provision and revocation of complex
network infrastructure and
resources in an instant by (re)programming has made it possible
for clouds to scale.
Cloud computing is a paradigm shift where we have a large pool
of computing
resources working together or separately depending upon the
requirements. Cloud
users are allocated resources without needing to know the
underlying details of how
123
Instance launch-time analysis of OpenStack virtualization… 993

56. that is achieved (unlike distributed computing where resources
allocation is more
simplistic, and in some cases, even requires occasional human
intervention).
2.4 OpenStack CMS
The OpenStack cloud management system (CMS) provides IaaS
composed of multiple
components that perform specific tasks. OpenStack is an open
source project, which
means that its use is free of cost, unlike the cloud systems
already available such as
Amazon EC2, Rackspace, HP, etc. Communication between
OpenStack components
is handled via the advanced message queuing protocol (AMQP)
and a REST API.
Key services provided by different OpenStack components
include compute (Nova),
image storage (Glance), storage (Swift), networking (Neutron)
and authentication
(Keystone), among many other optional components. Nova is a
core component of
Openstack that performs creation and termination of VMs. It
communicates with other
OpenStack components through the nova-api. Glance stores OS
images used to
boot various instances. Swift is an object storage service and is
responsible for storing
user data on the OpenStack cloud. Neutron, the networking
component, provides
network virtualization as a service. Virtual networks built from
virtual links, virtual
switches, virtual routers and other virtual network elements,
function the same way as

57. an equivalent physical network infrastructure. Finally, all users
accessing OpenStack
resources, and all their requests, need to be authenticated. This
is the function of
Keystone, i.e., to generate authentication tokens and keypairs to
authenticate users
and access requests for resources (Fig. 1).
2.5 OpenStack virtualization technologies
OpenStack’s Nova component supports different compute
virtualization technologies,
including hypervisors and container technologies. The three
Nova compute virtual-
ization technologies considered in this study include KVM,
LXD and Docker are
explained in Fig. 2.
2.5.1 Docker
Docker is open source container management software. It
enables the deployment and
bundling of applications inside a software container. Like VMs,
Docker containers
are portable and can be exported from one system to another. A
container does not
require a full operating system. Docker uses cgroups and kernel
namespaces to iso-
late applications running in containers and avoids the overhead
of booting a separate
operating system and its management overhead. It has a self
contained filesystem and
base image. Isolation gives the impression that the container is
running like a single
process on the host. Docker uses the libcontainer library to
communicate with

58. the kernel. The nova-docker driver is a hypervisor driver for
OpenStack Nova and has
been extended to spawn Docker containers. In order to spawn
containers, the Nova
driver is pointed to the Docker driver. The nova-docker driver
talks to the Docker agent
123
994 J. Ahmed et al.
Fig. 1 OpenStack architecture [27]
Fig. 2 Hypervisor VM versus LXD versus Docker [32]
123
Instance launch-time analysis of OpenStack virtualization… 995
using HTTP API calls. Docker images are stored in the Docker
registry and images
are exported to Glance from the Docker registry which Nova
uses to create containers.
2.5.2 Nova-LXD
LXD is container hypervisor and is only available on Linux. It
comprises of three
components:
– System-wide daemon (lxd)
– Command line client (lxc)

59. – OpenStack Nova LXD plugin (nova-compute-lxd).
LXD are machine containers, also called infrastructure
containers that run a full
Linux system. The same management and deployment tools used
for VMs and phys-
ical machines can also be used for LXD containers. In contrast,
Docker supports
stateless, minimal containers that cannot be upgraded or re-
configured but instead just
be replaced entirely. This makes Docker a software mechanism
rather than a machine
management tool and are hence named application containers.
LXDs are full Linux
systems, whereas Docker can be installed inside LXD container
to run applications or
other container applications.
2.6 Cloud benchmark suites
Benchmarking suites evaluate system performance for certain
tasks or workloads and
the results are compared against some predefined/standard
metrics. Several benchmark
suites, some specifically for OpenStack, are available for
performance testing of cloud
environments [3]. Sobel et al. [35] developed Cloudstone, a
CMS agnostic benchmark-
ing suite demonstrated on Amazon EC2 that aims to simulate
loads more characteristic
of Web 2.0 applications, consisting of database queries and user
queries. Luo et al. [21]
is another cloud benchmark suite that performs tests of typical
of (big) data processing
applications. Netflix, a commercial video streaming service, is
hosted on Amazon’s

60. cloud infrastructure. Netflix developed its own test suite called
Chaos Monkey that
deliberately introduces a variety of faults into the system (e.g.,
randomly shutting off
VMs, disks, network interface cards, etc.) and assesses the
service’ fragility in the face
of such failures [37]. Jackson et al. [17] measured the
performance of Amazon EC2
and compare it with that of a cluster using the same benchmarks
routinely used to test
high performance computing systems. The variation in
measurements from Amazon
EC2 were considerably higher, which was confirmed by several
other studies [30].
Schad [33] studied performance consistency of Amazon EC2
using the LINPACK
benchmark and concluded that variance of measurements is very
high.
Cloud computing is prevalent, but service failures are inevitable
and still occur
from time to time. The massive hardware and the immense task
of managing it can
result failures which can lead to service outages. During its
early days, Osterman et
al. [28] also conducted a performance study of Amazon EC2
with an eye on scientific
computing applications. They concluded that cloud computing
still required an order
of magnitude improvement before it could become useful for
scientific applications.
123

61. 996 J. Ahmed et al.
The wide deployment of cloud services across organizations has
brought about a need
for benchmarking clouds.
Amazon is among the earliest cloud service providers and has
been the subject of
several benchmarking studies including Schad et al. [33], Iosup
et al. [16], Jackson et
al. [17], Yigitbasi et al. [40], and O’Loughlin and Gillam [26].
A few cloud specific
benchmarks are available in literature Folkerts et al. [8],
Binning et al. [2] and Rak et
al. [29].
The performance of similar cloud systems may diverge
significantly. Li et al. [20]
developed CloudProphet, a systematic end-to-end benchmark
suite that helps cus-
tomers better predict an application’s running time when it is
moved to a cloud.
Variations are observed in virtual instances, storage services,
and network transfers
against some specific metrics for four different clouds which
are dominating the mar-
ket. While dealing with clouds, getting comparable results from
a cloud benchmark
can be deceptively difficult, due to differences between
measures, networks, and the
definitions of benchmarks. More recently, Google launched
PerfKit [10]. Perfkit cal-
culates end-to-end time to provision resources in clouds and can
be run on Amazon
AWS, Microsoft Azure and Google Cloud platforms.

62. 2.7 Instance launch time
Dynamic scalability of cloud is only meaningful when
additional/replacement
resources are available in a timely manner. Instance launch time
is an important
parameter for situations during service upscaling and migration,
particularly for time-
sensitive applications. Several recent studies have focused on
instance launch time,
e.g., Osterman et al. [28] measured launch time of single and
multiple instances on
Amazon EC2. Hill et al. [14] analyzed launch time for WebRole
and WorkRole on
the Microsoft Azure cloud. Mao et al. [23] studied the
relationship between instance
launch time and other factors such as type of instance acquired.
In contrast, we have
studied the effect of network conditions resulting from
temporary (excess load) or
permanent (faulty network equipment) on instance launch time.
Steinmetz et al. [36]
compared performance test results of OpenStack and Eucalyptus
cloud platforms.
Although they used instance launch time as a metric, their cloud
infrastructure con-
sisted of only two computers: a 16 core server and a Pentium 4
PC. Furthermore, the
work does not consider scalable deployment or any fault
injection mechanism to test
performance.
3 Experimental methodology
Cloud computing enables users to scale up or scale down
resources based on workload

63. and application requirements, i.e., users can acquire more
virtual resources as needed
during periods of high demand, and release virtual resources
during periods of low
demand. Such dynamic provisioning of virtual resources is
meaningful only when
it can be achieved fast enough. An instance can be requested at
any time by a user,
however, it may take up to several minutes for it to become
ready for use. This is the
time it takes the CMS to match the resource request to available
physical resources
123
Instance launch-time analysis of OpenStack virtualization… 997
and allocate them. The launch time may vary depending on a
combination of factors
including, but not limited to, type and size of image, copy/boot
of OS image, number
of cores, etc.
3.1 OpenStack infrastructure
The OpenStack IaaS model includes compute (Nova), image
storage (Glance), per-
sistent storage (Swift), networking (Neutron) and authentication
(Keystone) services.
The OpenStack project was first released in 2010, through a
collaboration between
RackspaceandNASA.TheOpenStackdevelopmentcommunitylaun
chesnewreleases
every six months with specific milestones. We deployed the

64. stable Liberty release, the
12th in the line.
OpenStack is highly scalable, meaning it can be deployed on as
few as a single and
as many as hundreds of nodes. In single node deployment, there
is only one computer
which hosts all OpenStack services, making it a complete cloud.
On a multi-node
OpenStack deployment, services are distributed across two or
more nodes, usually
where one node acts as a controller node and all the other nodes
are compute nodes.
– The controller node runs all the core services of OpenStack,
and is a key element
of the control plane of OpenStack environment.
– The compute node, on the other hand, is a collection of
services which runs the
hypervisor portion of compute.
It receives requests from the controller node to
allocate/deallocate instances.
The cloud scales horizontally by increasing the number of
compute nodes.
All these nodes are connected to an internal network (or
management network), as
shown in Fig. 3. An internal network is a separate network used
by cloud providers
consisting of separate switches, network and NICs, and provides
internal commu-
nication between the OpenStack components. This segregated
network is reachable
only from within a data center and prevents service disruption
by traffic generated by

65. tenants.
3.1.1 Instance launch sequence
In order to measure the instance launch time in OpenStack, it is
very important to
understand the sequence of steps involved. Various OpenStack
components interact
with a series of inter-component requests to successfully launch
an instance as shown
in Fig. 4. We are going to present simplistic steps involved in
the process, a more
elaborate description is already covered in [22]. All OpenStack
components commu-
nicate with each other using REST while the intra-service
communication is through
the remote procedure calls (RPCs) using relevant APIs. The
instance launch request
is made via CLI or Dashboard and it is translated to the nova-
boot command by the
Nova API server. Nova-api service interacts with Keystone for
authentication (1, 2).
Following successful authentication nova-db service is used to
create the initial entry
for the new instance (4, 5, 6, 7). The nova-api then interacts
with nova-scheduler to
get information of host where the instance has to be launched
(8, 9). After filtering and
123
998 J. Ahmed et al.
Fig. 3 Multi-node deployment

66. weighing, the nova-scheduler selects an appropriate host and
send a launch request to
nova-compute (10, 11, 12, 13). The nova-compute then makes
an RPC call to nova-
conductor for fetching information such as host ID, flavor, disk
and vCPU (14, 15, 16
17, 18). Nova-compute then makes a REST call to glance to
retrieve and upload the
image from the image database, to the selected host (19, 20).
This uploaded image is
cached for future use. Subsequently, nova-compute calls the
neutron to retrieve net-
work allocation and configuration information so that a fixed IP
is assigned to the new
instance (21, 22). Finally, nova-compute forwards all
information to the virtualization
driver, which executes the request of spawning an instance on
the hypervisor (23). It
is worth mentioning here that the volume storage backend (i.e.
Cinder or Ceph) is not
enabled in our experimental setup, therefore we have skipped
the steps involved in con-
tacting the cinder during the whole process. The corresponding
instance states visible
during the various stages of the provisioning process are:
Scheduling ¿ Networking ¿
Spawning ¿ Running [12].
3.2 Testing infrastructure
We have set up a control plane network with a single subnet for
communication
between all OpenStack nodes. The testing infrastructure
consisted of one controller
node with the following specifications: Intel Core i5 3.2GHz,

67. 10GB RAM, and a
500GB, 7200rpm SATA hard disk. We also set up three compute
nodes, each with
the same specifications as the controller node described above.
OpenStack relies
123
Instance launch-time analysis of OpenStack virtualization… 999
Fig. 4 Instance launch sequence
on inter-service communication between different nodes for
proper operation. This
communication requires the network to be fault-resilient.
Considering how critical
inter-service communication is to the proper functioning of
OpenStack, our fault injec-
tion mechanism targets service communications (on the control
plane network).
3.3 Fault injection mechanism
Considering the critical nature of inter-service communication
and the various types of
network errors inherent to computer networks, our fault
injection mechanism targets
inter-service communication on the control plane network. In
the OpenStack setup
shown in the Fig. 3, this is marked as the Management network.
The management
network in our case is just carrying the inter-service
communication traffic, and not
the VM traffic, to isolate test-case scenario we have

68. represented. We used the tc
qdisc utility in Linux [15] to simulate various network errors
and control their degree
of severity. tc qdisc consumes very little system resources and,
therefore, has a
negligible effect on measurements, making it a good
approximation of actual network
failures. We used tc qdisc to induce three kinds of faults in the
management
subnet: (1) limited bandwidth, (2) link delay, and (3) packet
losses, using the following
command lines:
– To limit the bandwidth to 100kbps: sudo tc qdisc add dev eth1
root tbf rate 100kbit burst 1600 limit 3000.Heretheburst
parameter specifies the buffer or max burst size of the bucket
(bytes per cell). The
limit parameter specifies the number of bytes that can be
queued waiting for
tokens to become available.
123
1000 J. Ahmed et al.
– To add a delay of 100ms: sudo tc qdisc add dev eth1 root
netem
delay 100ms.
– To set the packet loss rate on the link to 10%: sudo tc qdisc
add dev
eth1 root netem loss 10%.
– To remove all modifications made by tc qdisc we called, sudo

69. tc qdisc
del dev eth1 root.
These tc qdisc commands take effect at the physical NIC of the
host machines
of the testbed (see Fig. 5).
3.3.1 Packet loss
Packet losses can have noticeable effects on communication
network. We increased
packet loss rate in a controlled manner from 0% in uniform
intervals up until instances
failed to launch altogether.
3.3.2 Bandwidth
Bandwidthisanotherimportantparametertoconsiderwhenshapingt
henetworktraffic
between the nodes. Bandwidth was reduced by uniform step size
until the instances
were failed to launch successfully.
Fig. 5 System under test: turn around time
123
Instance launch-time analysis of OpenStack virtualization…
1001
3.3.3 Delay
Finally, we consider various levels of delay on the control plane
network. We begin by

70. adding no delay, then gradually increase it by uniform
increments up until instances
fail to launch.
3.4 Performance metric
The purpose of this experiment is to measure the time it takes to
launch varying sizes
of batches of VMs, LXD and Docker container instances. In
real-world scenarios
these launch times are of critical importance while scaling out a
VNF or meeting the
resource demand during peak hours of traffic etc. For this
purpose, we developed a
Python script using the python-novaclient bindings of
OpenStack. This script
repeatedly executes a series of tests. Each test consists of
launching a batch of varying
number of VMs, LXD and Docker containers with image pre-
cached on compute node
prior to tests.
Next, measuring the time it takes for all instances to become
active. An instance
is considered to have reached the active state as soon as it
becomes accessible via its
virtual NIC. Measuring the launch time is of prime importance
because it can capture
system’s overall performance. We use the time it takes to
receive a successful ping
response as a proxy indicator that an instance has launched
because in OpenStack the
time it takes for an instance to go from active state to ready to
use state is negligible.
An instance is in active state if there are no ongoing compute
API calls (running

71. tasks) while instance is ready to use when operating system
boots up and all internal
infrastructure components like networks and volumes are
attached and ready for used.
Finally, the script terminates all instances to return the system
to its initial state
ahead of the next test. For consistency of results all the KVM
instances are launched
using the Ubuntu 14.04 cloud image and m1.small flavor.2. All
Docker containers are
launched using the Ubuntu 14.04 base image from Docker
registry, while the LXD
containers are spawned using Ubuntu 14.04 image. The
difference in the launch time
of a VM as compared to the container, which we will see in the
next section, can be
visualized by the turn around of the ping in both these cases as
shown in Fig. 5. In the
case of VM, the ping packet has to traverse through a guest OS
layer while in the case
of containers there is no additional OS layer.
When we were designing this series of experiments we
encountered situations
where instances launch requests would end up failing. After
thorough exploration we
observed that the longest it took any one or group of instances
to launch was 450s. Fur-
ther deterioration in networking conditions would result in
complete failure to launch.
Even waiting for up to 30min would not yield a successful
launch. This series of exper-
iments is repeated for various levels of packet losses, delay and
bandwidth limitations.

72. 2 Flavor is the term used in OpenStack documentation to refer
to various configurations of VM instances.
123
1002 J. Ahmed et al.
100 120 140 160 180 200 220 240 260 280 300
Bandwidth (kbps)
0
20
40
60
80
100
120
140
160
180
200
S

73. pa
w
ni
ng
ti
m
e
(s
ec
)
Bandwidth vs Spawning time
1 Docker Container
10 Docker Containers
20 Docker Containers
30 Docker Containers
40 Docker Containers
50 Docker Containers
Fig. 6 Spawning time versus bandwidth for Docker containers
100 120 140 160 180 200 220 240 260 280 300
Bandwidth (kbps)
0
20
40

74. 60
80
100
120
140
160
180
200
S
pa
w
ni
ng
ti
m
e
(s
ec
)
Bandwidth vs Spawning time

75. 1 LXD Container
10 LXD Containers
20 LXD Containers
30 LXD Containers
40 LXD Containers
50 LXD Containers
Fig. 7 Spawning time versus bandwidth for LXD containers
4 Results and analysis
4.1 Launch time versus fault levels
4.1.1 Variable bandwidth
Figures 6, 7 and 8 depict the average launch times for Docker
containers, LXDs
and VMs, respectively, at various bandwidth levels in the
control plane network.
Note that all plotted data points are average launch times of 10
runs. All three plots
123
Instance launch-time analysis of OpenStack virtualization…
1003
120 140 160 180 200 220 240 260 280 300
Bandwidth (kbps)
0
50

76. 100
150
200
250
S
pa
w
ni
ng
ti
m
e
(s
ec
)
Bandwidth vs VM Spawning time
1VM
5VMs
10VMs
Fig. 8 Spawning time versus bandwidth for VMs
in this group are for bandwidth ranging from 90 to 300kbps
depending upon the

77. different virtualization technologies used. It is worth noticing
that LXD containers
are performing slightly better than Docker containers and VMs
because the threshold
value of bandwidth for LXD container is as little as 90kbps,
while it is 100kbps for
Docker containers and 110kbps for VMs. Below these
bandwidth thresholds, these
virtualization technologies failed to launch their specific
instances. Not surprisingly,
the figures also shows that the launch times increase with larger
batch sizes for all
three virtualization technologies.
We also observed that for similar parameters (batch size and
bandwidth), LXD
offers faster launch times than Docker containers and much
faster launch times than
VMs.
Furthermore, we observe that for each virtualization technology
launch times
increase linearly with decrease in bandwidth, until an inflexion
point is reached beyond
which launch times increase at a much higher rate. For Docker
containers that inflexion
point appears at 105kbps, and for LXD at 115kbps.
Interestingly, the inflexion point
remains the same regardless of the batch size. This inflexion
point is less prominent
in the case of VMs. However, for the 10 VM case there appears
to be greater increase
in launch times for bandwidths less than 185kbps.
Most obviously and unsurprisingly, the same testing
infrastructure was able to

78. launch LXD/Docker containers approximately an order of
magnitude faster than VMs.
This is explained by the smaller resource overhead of containers
relative to VMs.
4.1.2 Variable delay
Figures 9, 10 and 11 depict the average launch times for Docker
containers, LXDs and
VMs, respectively, at various delays in the control plane
network. Note that as before
all plotted data points are average launch times of 10 runs. All
three plots in this group
123
1004 J. Ahmed et al.
Delay (sec)
0
50
100
150
200
250
300

79. S
pa
w
ni
ng
ti
m
e
(s
ec
)
Delay vs Spawning time
1 Docker Container
10 Docker Containers
20 Docker Containers
30 Docker Containers
40 Docker Containers
50 Docker Containers
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Fig. 9 Spawning time versus delay for Docker containers
0 1 2 3 5
Delay (sec)
0
50

80. 100
150
200
250
300
350
400
450
S
pa
w
ni
ng
ti
m
e
(s
ec
)
Delay vs Spawning time

81. 1 LXD Container
10 LXD Containers
20 LXD Containers
30 LXD Containers
40 LXD Containers
50 LXD Containers
4
Fig. 10 Spawning time versus delay for LXD containers
are for delay ranging from 0 to 5–5.3s depending upon the
different virtualization
technologies used. It is worth noticing that LXD containers are
again performing
slightly better than Docker containers and VMs because the
threshold value of delay
at which LXD containers still manage to launch is as much as
5.3s while it is 5.2s for
Docker containers and 5s for VMs. Above these delay
thresholds, these virtualization
technologies failed to launch their specific instances. Not
surprisingly, the figures also
shows that the launch times increase with larger batch sizes for
all three virtualization
technologies.
123
Instance launch-time analysis of OpenStack virtualization…
1005
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

82. Delay (sec)
0
50
100
150
200
250
300
350
400
450
S
pa
w
ni
ng
ti
m
e
(s
ec

83. )
Delay vs VM Spawning time
1VM
5VMs
10VMs
Fig. 11 Spawning time versus delay for VMs
Packet loss rate
0
50
100
150
200
S
pa
w
ni
ng
ti
m
e
(s

84. ec
)
Packet loss vs Spawning time
1 Docker Conatiner
10 Docker Containers
20 Docker Containers
30 Docker Containers
40 Docker Containers
50 Docker Containers
0 0.1 0.2 0.3 0.4 0.5 0.6
Fig. 12 Spawning time versus packet loss rate for Docker
containers
We also tried to identify inflexion points on the delay axis
beyond which launch
times began increasing at a higher rate. For LXDs and VMs that
inflexion point appears
at around the 5s delay level. However, we do not see a clear
inflexion point for the
corresponding plot for Docker containers in Fig. 9.
In absolute terms, the launch times of same batch sizes of
Docker containers and
LXDs are about the same. However, launch times of equally
sized VMs are about one
order of magnitude higher.
123

85. 1006 J. Ahmed et al.
0 0.1 0.2 0.3 0.4 0.5 0.6
Packet loss rate
0
20
40
60
80
100
120
140
160
S
pa
w
ni
ng
ti
m
e

86. (s
ec
)
Packet loss vs Spawning time
1 LXD Container
10 LXD Containers
20 LXD Containers
30 LXD Containers
40 LXD Containers
50 LXD Containers
Fig. 13 Spawning time versus packet loss rate for LXD
containers
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
Packet loss rate
0
50
100
150
S
pa
w
ni
ng

87. ti
m
e
(s
ec
)
Packet loss vs VM Spawning time
1VM
5VMs
10VMs
Fig. 14 Spawning time versus packet loss rate for VMs
4.1.3 Variable PLR
Figures 12, 13 and 14 depict the average launch times for
Docker containers, LXDs
and VMs, respectively, at various packet loss rates (PLR) in the
control plane network.
Note that as before, all plotted data points are average launch
times of 10 runs. All
three plots in this group are for PLR ranging from 0 to 50–60%
depending upon the
different virtualization technologies used. Once again, Docker
containers and LXD
are performing better than VMs because the maximum threshold
of PLR at which
Docker containers and LXD are still able to launch is 60%,
while VMs were not
123

88. Instance launch-time analysis of OpenStack virtualization…
1007
No of Docker Containers
0
20
40
60
80
100
120
140
160
180
200
S
pa
w
ni

89. ng
ti
m
e
(s
ec
)
Effect of bandwidth
300kbps
250kbps
200kbps
150kbps
100kbps
0 5 10 15 20 25 30 35 40 45 50
Fig. 15 Spawning time versus No of Docker containers
able to launch when the PLR exceeded 50%. Above these PLR
thresholds, these
virtualization technologies failed to launch their specific
instances. Not surprisingly,
the figures also shows that the launch times increase with larger
batch sizes for all
three virtualization technologies.
We also observed that for similar parameters (batch size and
PLR), LXD offers
faster launch times than Docker containers and much faster
launch times than VMs.

90. Furthermore, we observe that for each virtualization technology
launch times
increase linearly with increase in PLR, until an inflexion point
is reached beyond
which launch times grow at a much faster rate. For Docker
containers and LXD that
inflexion point appears at about 45%, regardless of batch size.
This inflexion point is
less prominent in the case of VMs. However, for the 10 VM
case there appears to be
greater increase in launch times for PLRs above 35%.
4.2 Spawning time versus instance batch size
4.2.1 Effect of bandwidth
Figures 15, 16 and 17 are plots of average launch versus batch
sizes for a variety
of bandwidths for Docker containers, LXD and VMs,
respectively. These plots also
show what all the previous figures could not, i.e., if the
relationship between batch size
and launch time is linear or not. Figures 15 and 16 show that
launch times increase
roughly linearly with number of Docker containers and LXD but
rises faster than that
for (A) batch sizes greater than 40 Docker containers/LXDs, and
(B) bandwidth less
than 150kbps for Docker containers/LXDs. Figure 17 is a
similar plot for VMs and
shows that launch times remain linear from 1 to 10 VMs at all
bandwidths.
123

91. 1008 J. Ahmed et al.
No of LXD containers
0
20
40
60
80
100
120
140
160
S
pa
w
ni
ng
ti
m
e
(s

92. ec
)
Effect of bandwidth
300kbps
250kbps
200kbps
150kbps
100kbps
0 5 10 15 20 25 30 35 40 45 50
Fig. 16 Spawning time versus No of LXD containers
1 2 3 4 5 6 7 8 9 10
No of VMs
0
50
100
150
200
250
S
pa
w

93. ni
ng
ti
m
e
(s
ec
)
Effect of bandwidth
300kbps
250kbps
200kbps
150kbps
110kbps
Fig. 17 Spawning time versus No of VMs
4.2.2 Effect of delay
Figures 18, 19 and 20 plot average launch times versus batch
sizes for different delay
values for Docker containers, LXDs and VMs, respectively. We
observe that launch
times increase linearly at first, but as the batch size increases
beyond a threshold value
(which is different for different delays) the launch times grow
at a faster rate. This is
more clearly visible for Docker containers (Fig. 18) and LXD
(Fig. 19), but less so for
the case of VMs. For the cases of Docker and LXD containers,

94. for batch sizes of 50
instances, each 1s delay increases the launch time by
approximately 50s.
123
Instance launch-time analysis of OpenStack virtualization…
1009
No of Docker Containers
0
50
100
150
200
250
300
S
pa
w
ni
ng
ti

95. m
e
(s
ec
)
Effect of delay
0 sec
1 sec
2 sec
3 sec
4 sec
5 sec
0 5 10 15 20 25 30 35 40 45 50
Fig. 18 Spawning time versus No of Docker containers
No of LXD containers
0
50
100
150
200
250
300

96. 350
400
450
S
pa
w
ni
ng
ti
m
e
(s
ec
)
Effect of delay
0 sec
1 sec
2 sec
3 sec
4 sec
5 sec
5.6 sec
0 5 10 15 20 25 30 35 40 45 50
Fig. 19 Spawning time versus No of LXD containers

97. 4.2.3 Effect of PLR
Figures 21, 22 and 23 plot the average launch times versus
batch sizes for different
delay values for Docker containers, LXDs and VMs,
respectively. The PLR was varied
over a range of 0–60%, with launch times generally increasing
with increasing PLR.
Launch times of batches of Docker containers and LXDs grow
linearly up to about
30 instances for PLRs up to 40%. For PLRs more than that the
launch times remain
linear for batches of only about 20 instances. For the case of
VMs launch times in Fig.
23 appear to remain linear for all ranges of PLR and batch sizes.
123
1010 J. Ahmed et al.
1 2 3 4 5 6 7 8 9 10
No of VMs
0
50
100
150
200

98. 250
300
350
400
450
S
pa
w
ni
ng
ti
m
e
(s
ec
)
Effect of delay
0 sec
1 sec
2 sec
3 sec
4 sec
5 sec

99. 5.4 sec
Fig. 20 Spawning time versus No of VMs
0 5 10 15 20 25 30 35 40 45 50
No of Docker Containers
0
50
100
150
200
S
pa
w
ni
ng
ti
m
e
(s
ec
)
Effect of % packet-loss

100. 0% packet-loss
10% packet-loss
20% packet-loss
30% packet-loss
40% packet-loss
50% packet-loss
60% packet-loss
Fig. 21 Spawning time versus No of Docker containers
5 Conclusions
In this measurement study, we evaluated the ability of three
OpenStack virtualization
technologies, KVM, LXD and Docker containers, to continue to
provide useful ser-
vices in the face of network errors (limited bandwidth, delay
and packet losses) of
varying degrees of severity in the control plane network. This
performance analysis
was performed using OpenStack’s 12th release, named Liberty.
Overall, for most tests LXD exhibited the fastest launch times,
followed very closely
by Docker containers. We also observed that OpenStack clouds
is generally able to
123
Instance launch-time analysis of OpenStack virtualization…
1011
No of LXD containers

101. 0
20
40
60
80
100
120
140
160
S
pa
w
ni
ng
ti
m
e
(s
ec
)

102. Effect of % packet-loss
0% packet-loss
10% packet-loss
20% packet-loss
30% packet-loss
40% packet-loss
50% packet-loss
60% packet-loss
0 5 10 15 20 25 30 35 40 45 50
Fig. 22 Spawning time versus No of LXD containers
1 2 3 4 5 6 7 8 9 10
No of VMs
0
50
100
150
S
pa
w
ni
ng
ti
m

103. e
(s
ec
)
Effect of % packet-loss
0% packet-loss
10% packet-loss
20% packet-loss
30% packet-loss
40% packet-loss
50% packet-loss
Fig. 23 Spawning time versus No of VMs
launch an order of magnitude more container instances on the
same infrastructure than
VMs.Intermsoflimitedbandwidth,delayandPLRweconsistentlyobs
ervedcontainer
virtualization technologies to be a little more resilient than
VMs. Our measurement
results show that there is a lower limit on the bandwidth, i.e.,
approximately 110kbps
on the control plane network, below which instances will not
launch. Differences
between virtualization technologies in this regard were very
slight. In terms of delay,
containers can bear a delay of up to 5.5s, while KVM continues
working for up to 5s.
In terms of packet-losses, containers continue to launch even
when the PLR is as high
as 60%, while KVM VMs can only weather PLRs up to 50%.

104. 123
1012 J. Ahmed et al.
We also observed various batch sizes, specific to our test
environment, for which
instance launch times grow linearly with batch size, depending
on the severity level
of network errors.
In conclusion, we would like to acknowledge that all results
presented in this paper
were produced from experiments conducted on an OpenStack
testbed comprising of
four physical host machines. We realize that the size of this
setup is not representative
of a production environment data center. Although all reported
results are the averages
of multiple runs, future studies investigating the effects of
network errors on instance
launch times should be conducted on larger, more representative
size testbeds.
References
1. Akioka S, Muraoka Y (2010) HPC benchmarks on Amazon
EC2. In: 2010 IEEE 24th international
conference on advanced information networking and
applications workshops (WAINA). IEEE, pp
1029–1034
2. Binnig C, Kossmann D, Kraska T, Loesing S (2009) How is
the weather tomorrow?: towards a bench-
mark for the cloud. In: Proceedings of the second international

105. workshop on testing database systems.
ACM, p 9
3. Cooper BF, Silberstein A, Tam E, Ramakrishnan R, Sears R
(2010) Benchmarking cloud serving
systems with YCSB. In: Proceedings of the 1st ACM
symposium on Cloud computing. ACM, pp
143–154
4. Costa P, Migliavacca M, Pietzuch P, Wolf AL (2012) Naas:
network-as-a-service in the cloud. In:
Proceedings of the 2nd USENIX conference on Hot Topics in
Management of Internet, Cloud, and
Enterprise Networks and Services, Hot-ICE, vol 12, p 1
5. Erickson D (2012) Floodlight java based openflow controller.
Last accessed, Ago
6. Erickson D (2013) The beacon openflow controller. In:
Proceedings of the second ACM SIGCOMM
workshop on Hot topics in software defined networking. ACM,
pp 13–18
7. Feamster N, Rexford J, Zegura E (2013) The road to SDN: an
intellectual history of programmable
networks
8. Folkerts E, Alexandrov A, Sachs K, Iosup A, Markl V, Tosun
C (2013) Benchmarking in the cloud:
what it should, can, and cannot be. In: Selected topics in
performance evaluation and benchmarking.
Springer, pp 173–188
9. Garfinkel SL (2007) An evaluation of Amazon’s grid
computing services: EC2, S3, and SQS. In: Center
for. Citeseer