Brighttalk learning to cook- network management recipes - final

Learning to Cook:
Network Management Recipes
https://cbsstlouis.files.wordpress.com/2013/01/kidscooking.jpg

(C) SystemsManagementZen.com 2007-2015. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Mr. White has over ﬁfteen years of experience designing and managing
the deployment of Systems Monitoring and Event Management software.
Currently, he is serving as the Operational Readiness Leader for a Fortune
50 Enterprise. Mr. White has also held positions including Executive
Architect at IBM, leader of the Monitoring and Event Management
organization at Nationwide Insurance and owner of a Service
Management Consultancy developing solutions for a wide variety of
organizations, including the Mexican Secretaría de Hacienda
y Crédito Público, Telmex, Wal-Mart of Mexico, JP Morgan Chase,
Nationwide Insurance and the US Navy Facilities and Engineering
Command.
Andrew White
Long Time System Management Expert
UX Evangelist

For those of you who are
sleeping right now…

This topic isn’t going to help much.
SORRY :(

http://weheartit.com/entry/12433848

Ground rules for this
session…
•  If you can’t tell if I am trying to be funny…
– 
GO AHEAD AND LAUGH!
•  Feel free to text, tweet, yammer, or whatever.
Use
•  If you have a question, no need to wait until
the end. Just interrupt me. Seriously… I
don’t mind.

I have a lot of experience leading
Systems and Event Management teams

Latency
I am here today to share some of what I have learned about

User
Experience
And more importantly, I am here today to talk about

What do I mean by latency and user experience?

LaŸtenŸcy – [LEYT-n-see]
-noun, plural -cies
1.  The state of being latent
2.  The time that elapses between a stimulus and the
response to it
3.  The state of being not yet evident or active

http://www.flickr.com/photos/25822731@N02/4644128723/

ExŸpeŸriŸence – [ik-SPEER-ee-uh’ns]
-noun
1.  The apprehension of an object, thought, or emotion through
the senses or mind
2.  Direct personal participation or observation; actual knowledge
or contact
3.  A particular incident, feeling, etc., that a person has
undergone
-verb
4.  To be emotionally or aesthetically moved by; to feel
5.  To learn by perceiving, understanding, or remembering

http://www.ﬂickr.com/photos/51035626620@N01/170061976/sizes/l/in/photostream/

When you put them together we get:

The ultimate measure of success for any system is
the perception of its performance. The less
interactive a system becomes the more likely its
performance will be perceived to be poor.
Latency is the mother of inactivity!

The Two Dimensions of
Latency…
Internal Latency vs. External Latency
Actual Latency vs. Perceived Latency
This is what user experience is all about
In other words: Perceived = Fn(Internal+External)Variation )

We need to recognize when we
have problems to solve

Maybe.
Let me show you why this is important…

Start…
Start…
Observed Maximum:
90th Percentile:
5.44 seconds…
15.4 seconds…

Start…
Start…
Observed Maximum:
90th Percentile:
DONE!5.44 seconds…
15.4 seconds…

Start…
Start…
Observed Maximum:
90th Percentile:
DONE!
DONE!
5.44 seconds…
15.4 seconds…

If you were the one on the phone
with one of those customers…
how would you fill that silence?

No complaint… is more common
than that of a scarcity of money
-Adam Smith, Wealth of Nations

*Among adults who accessed the internet with a mobile phone in the past 12 months (n=1,001) – Gomez Mobile Web Experience Survey conducted by Equation Research
58% of mobile phone users expect websites
to load as quickly, almost as quickly or faster
on their mobile phone, compared to the
computer they use at home*
http://www.ﬂickr.com/photos/lucianbickerton/3858380291/sizes/l/

*Among adults who accessed the internet with a mobile phone in the past 12 months (n=1,001) – Gomez Mobile Web Experience Survey conducted by Equation Research
60% of mobile web users have had a problem in the
past year when accessing a website on their phone*
http://www.ﬂickr.com/photos/rickyromero/1357938629/sizes/l/

*Among adults who accessed the internet with a mobile phone in the past 12 months (n=602) – Gomez Mobile Web Experience Survey conducted by Equation Research
Slow load time was the number on issue,
experience by almost 75% of them*
http://bighugelabs.com/onblack.php?id=2497744197&size=large

Our Problem Statement:
The business needs to reliably reach its customers and
users regardless of where they may be located. Latency
forces close geographic proximity of the components
and limits the quality of service provided to
geographically distributed customers.

If the users can’t use it, it
doesn’t work.

Our Constraints
At the same time, there are a few inescapable facts we face:
1.  Today’s users demand reliable systems to do their work
2.  IT systems will mirror the complexity of the businesses
they support
3.  Our environments must be massive to handle the workload
4.  Business continuity requires geographic diversity in our
deployment locations
5.  The speed of light isn’t changing any time soon

When all of these happen at the same time…
Ug…

Question
Is there a better way to ﬁgure out what
monitoring would help?

Itemize the
existing
monitors
Brainstorm
potential gaps
to ﬁll
Deploy new
monitors
Identify the
potential
risks
Itemize the
existing
monitors
Determine
if which
gaps exist
Fill the
monitoring
gaps
Current Approach
Proposed Approach
Picking Better Monitors

What Do You Want To Accomplish?
Your monitoring should help you answer:
•  How will we know if the users are getting the experience
they are expecting?
•  How much capacity do we need during normal and peak
times to ensure user expectations are met?
•  How quickly can the provider we select ramp up to meet
our needs if we ﬁnd that the service is underperforming?
•  How fast do we need to be able to access additional
capacity once it is ready for us?

Composite
Applications
Site Content
Search
Session
Information
User Login
& Identity Mgmt
Content Mgmt
System
Social Network
Widgets
Site Tracking
& Analytics
Banner Ads &
Revenue Generators
Multimedia &
CDN Content

Composite Applications Are
Everywhere
•  ATG (Oracle) – Shopping Cart
•  Estara – Click to Chat
•  Twitter Widget – Social Networking
•  Gigya – Social Networking
•  Google Maps API – GeoLocation
•  Facebook Widget – Social Networking
•  Google Analyics – User Tracking

Seeing Is Believing
Real User Monitoring
Would Report 94ms
Response Time.
The page seemed
“done” to me
1.2 seconds later
The time spent rendering
represented 93% of the
user experienced latency

The Same Old Problem
Corporate
LANs & VPNs
ISP
Connection
DNS & Internet
Services
Content Mgmt
System
Social Network
Widgets
Site Tracking
& Analytics
Banner Ads &
Revenue Generators
Multimedia &
CDN Content
Home Wireless
& Broadband
Mobile Broadband
Is It My Data Center?
•  Configuration errors
•  Application design issues
•  Code defects
•  Insufficient infrastructure
•  Oversubscription Issues
•  Poor routing optimization
•  Low cache hit rate
Is It a Service Provider Problem?
•  Non-optimized mobile content
•  Bad performance under load
•  Blocking content delivery
•  Incorrect geo-targeted content
Is it an ISP
Problem?
•  Peering problems
•  ISP Outages Is it My Code or a Browser Problem?
•  Missing content
•  Poorly performing JavaScript
•  Inconsistent CSS rendering
•  Browser/device incompatibility
•  Page size too big
•  Conflicting HTML tag support
•  Too many objects
•  Content not optimized for device
The Cloud
Distributed
Database
Mainframe
Network
Middleware
Storage

Cognitive Dissonance
Corporate
LANs & VPNs
Distributed
Database
Mainframe
Network
Middleware
Storage
ISP
Connection
DNS & Internet
Services
Content Mgmt
System
Social Network
Widgets
Site Tracking
& Analytics
Banner Ads &
Revenue Generators
Multimedia &
CDN Content
Home Wireless
& Broadband
Mobile Broadband
The Part You Control
The Part They Experience
…meanwhile
the user is
NOT
happy
All our systems
look great,
SLA’s are being
met…
You Have More
Control Here Than
You Think

Gaining Perspective
Requires Balance
Packet Capture
Synthetic Transactions
Client Monitoring
Client Monitoring
Synthetic Transactions
Server Probe
1.  Client to the Server
2.  Server to the Client
3.  “3rd Party” Vantage Point
4.  Synthetic Transactions
Four Perspectives of User Experience

Why Multiple Perspectives?
Know Your Customer:
•  What they do?
§  Customers care about completing tasks
NOT whether the homepage is available
•  Where they do it from?
§  Your customers don’t live in the cloud, test from their perspective
•  When they do it?
§  Test at peak and normal trafﬁc levels, to ﬁnd all the problems
•  What expectations do customers have?
§  Is 5 seconds fast enough or does it have to be quicker?

What Does Good
Monitoring Look Like?
Corporate
LANs & VPNs
Load Balancer
Load Balancer
Firewall
Switch
Web Server Farm
Database
Data Power
Mainframe
Middleware
Load Balancer
1.  System Availability
2.  Operating System Performance
3.  Hardware Monitoring
4.  Service/Daemon and Process Availability
5.  Error Logs
6.  Application Resource KPIs
7.  End-to-End Transactions
8.  Point of Failure Transactions
9.  Fail-Over Success
10. “Activity Monitors” and “Reverse Hockey Stick”
Elements of Good Monitoring
32 4 5 61
7
8
9 10

When decisions are not made based
on information, it’s called gambling.

Finding Metrics That Matter
§  Will the metric be used in a report? If so, which one? How is it used in the report?
§  Will the metric be used in a dashboard? If so, which one? How will it be used?
§  What action(s) will be taken if an alert is generated? Who are the actors? Will a ticket
be generated? If so, what severity?
§  How often is this event likely to occur? What is the impact if the event occurs? What
is the likelihood it can be detected by monitoring?
§  Will the metric help identify the source of a problem? Is it a coincident / symptomatic
indicator?
§  Is the metric always associated with a single problem? Could this metric become a
false indicator?
§  What is the impact if this goes undetected?
§  What is the lifespan for this metric? What is the potential for changes that may
reduce the efﬁcacy of the metric?
Evaluating the Effectiveness of a Metric

Watch your words
737-900ER 747-400ER
Maximum Number of
Passengers
215
524
Maximum Crusing Speed (mph)
511
570
A 737 and a 747 both travel around 500 mph but the 747
carries twice as many people. Would you say it is twice as fast?

What Matters Most?
Dr. Lee
Goldman
Cook County Hospital,
Chicago, IL
§  Is the patient feeling unstable
angina?
§  Is there ﬂuid in the patient’s lungs?
§  Is the patient’s systolic blood
pressure below 100?#

The Goldman Algorithm
Prediction of Patients Expected to
Have a Heart Attack Within 72 Hours
0

20

40

60

80

100

Traditional Techniques
Goldman Algorithm
By paying attention to what really matters, Dr.
Goldman improved the “false negatives” by 20
percentage points and eliminated the “false
positives” altogether.

•  Server Metrics
–  Server Response Time
–  Server Connection Time
–  Refused Session Percentage
–  Unresponsive Session Percentage
•  Network Metrics
–  Network Round Trip Time
–  Retransmission Delay
–  Effective Network Round Trip Time
–  Network Connection Time
•  Application Metrics
–  Total Transaction Time
–  Data Transfer Time
Really Helpful KPIs

Beware of Averages
75th
Percentile
50th
Percentile
25th
Percentile
0.5 0.7 0.9 1.8 2.5 2.5 2.6 2.9 3.3 3.5
Average

Your Mission…
In addition to monitoring for system availability, we are
here to help manage latency.
The Recipe:
1.  Continually map, monitor, and categorize all
sources of latency
2.  Help identify and remove all sources that are found

The Critical Path of
Performance
Browser
Workstation
OS
Workstation
Hardware
Client LAN
Corporate
WAN
Datacenter
LAN
Etc.
Web Server
Web Server
OS
Web Server
Hardware
Datacenter
LAN
Middleware
Server
Hardware
Middleware
Server OS
Middleware
Application
Etc.
Database
Server
Database
Server OS
Database
Server
HBA
SAN
Fabric
Switch
Array
Hardware
Array
Controller
Hardware
Cache
Disk
Drives
Etc.
Client Node
Middleware
Database

MIBs and OIDs
root
iso (1)
org (3)
dod (6)
Internet (1)
Interfaces (2)
IP (4)
System (1)
ifOperStatus = ..1.3.6.1.2.1.2.2.1.8.0
MIB-2 (1)
Directory (1)
Experimental (3)
Mgmt (2)
Private (4)
Juniper (2636)
Cisco (9)
Apple (63)
Microsoft (311)
Port OperStatus = .1.3.6.1.4.1.9.5.1.4.1.1.6.0Functionally the same

MIBs and OIDs
root
iso (1)
org (3)
dod (6)
Internet (1)
Interfaces (2)
IP (4)
System (1)
MIB-2 (1)
Directory (1)
Experimental (3)
Mgmt (2)
Private (4)
Juniper (2636)
Cisco (9)
Apple (63)
Microsoft (311)
Port Index = .1.3.6.1.4.1.9.5.1.4.1.1.4.0 A MIB is the set
of OIDs for a
defining a set of
information in the
database
Port Type = .1.3.6.1.4.1.9.5.1.4.1.1.5.0
Port OperStatus = .1.3.6.1.4.1.9.5.1.4.1.1.6.0
Port IfIndex = .1.3.6.1.4.1.9.5.1.4.1.1.11.0
portMacControlUnknownProtocolFrames = .1.3.6.1.4.1.9.5.1.4.1.1.21.0

RMON is “Flow-Based”
Monitoring
RMON v1 (RFC 2819)
•  Statistics: real-time LAN statistics e.g. utilization,
collisions, CRC errors
•  History: history of selected statistics
•  Alarm: definitions for RMON SNMP traps to be
sent when statistics exceed defined thresholds
•  Hosts: host specific LAN statistics e.g. bytes
sent/received, frames sent/received
•  Hosts top N: record of N most active
connections over a given time period
•  Matrix: the sent-received traffic matrix between
systems
•  Filter: defines packet data patterns of interest e.g.
MAC address or TCP port
•  Capture: collect and forward packets matching
the Filter
•  Event: send alerts (SNMP traps) for the Alarm
group
•  Token Ring: extensions specific to Token Ring
RMON v2 (RFC 4502)
•  Protocol Directory: list of protocols the probe can
monitor
•  Protocol Distribution: traffic statistics for each
protocol
•  Address Map: maps network-layer (IP) to MAC-
layer addresses
•  Network-Layer Host: layer 3 traffic statistics, per
each host
•  Network-Layer Matrix: layer 3 traffic statistics, per
source/destination pairs of hosts
•  Application-Layer Host: traffic statistics by
application protocol, per host
•  Application-Layer Matrix: traffic statistics by
application protocol, per source/destination pairs
of hosts
•  User History: periodic samples of user-specified
variables
•  Probe Configuration: remote configure of probes
•  RMON Conformance: requirements for RMON2
MIB conformance

The RMON MIBs
root
iso (1)
org (3)
dod (6)
Internet (1)
Interfaces (2)
IP (4)
System (1)
MIB-2 (1)
Directory (1)
Experimental (3)
Mgmt (2)
Private (4)
RMON (16)
RMON data is
stored in a MIB
and can be
collected using
SNMP

MIBs and OIDs
root
iso (1)
org (3)
dod (6)
Internet (1)
Interfaces (2)
IP (4)
System (1)
MIB-2 (1)
Directory (1)
Experimental (3)
Mgmt (2)
Private (4)
RMON (16)
rmonEventsV2
statistics
history
alarm
hosts
hostTopN
matrix
ﬁlter
Capture
Event
tokenRing
protocolDir
protocolDist
addressMao
nlHost
nlMatrix
alHost
alMatrix
usrHistory
probeConﬁg
rmonConformance
mediaIndependentStats
switchRMON
interfaceTopNMIB
hcAlarmMIB
=  .1.3.6.1.2.1.16.0
=  .1.3.6.1.2.1.16.1.0
=  .1.3.6.1.2.1.16.2.0
=  .1.3.6.1.2.1.16.3.0
=  .1.3.6.1.2.1.16.4.0
=  .1.3.6.1.2.1.16.5.0
=  .1.3.6.1.2.1.16.6.0
=  .1.3.6.1.2.1.16.7.0
=  .1.3.6.1.2.1.16.8.0
=  .1.3.6.1.2.1.16.9.0
=  .1.3.6.1.2.1.16.10.0
=  .1.3.6.1.2.1.16.11.0
=  .1.3.6.1.2.1.16.12.0
=  .1.3.6.1.2.1.16.13.0
=  .1.3.6.1.2.1.16.14.0
=  .1.3.6.1.2.1.16.15.0
=  .1.3.6.1.2.1.16.16.0
=  .1.3.6.1.2.1.16.17.0
=  .1.3.6.1.2.1.16.18.0
=  .1.3.6.1.2.1.16.19.0
=  .1.3.6.1.2.1.16.20.0
=  .1.3.6.1.2.1.16.21.0
=  .1.3.6.1.2.1.16.22.0
=  .1.3.6.1.2.1.16.23.0
=  .1.3.6.1.2.1.16.24.0
All this information lives in just
one table and most people
don’t know about it!

Setting Thresholds
Falling Threshold
Rising Threshold
Sample Interval
Policy Activations

How we view the network

How our applications view it

What a Flow Record Looks Like
http://www.cisco.com/c/en/us/td/docs/ios/fnetflow/configuration/guide/12_2sr/fnf_12_2_sr_book/fnetflow_overview.html

One record, multiple uses
http://www.cisco.com/c/en/us/td/docs/ios/fnetflow/configuration/guide/12_2sr/fnf_12_2_sr_book/fnetflow_overview.html

The Progression
SNMP
Granularity
Accuracy
RMON
Netﬂow
Packet
Inspection

That is great but we need more…

Shallow vs Deep Packet Inspection
SPI is very focused on header information from OSI Layers 3 & 4 (IP, TCP, UDP, etc.)
DPI processes header and datagram information (HTTP, SQL, SIP, etc.)
IP Header
TCP Header
GET /userLogin.jsp HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14
(KHTML, like Gecko) Version/7.0.3 Safari/7046A194A
Shallow Packet Inspection (SPI)
Deep Packet Inspection (DPI)

Shallow Packet Inspection

Degraded Threshold – The point at which users will
complain about poor performance

Excessive Threshold – The point at which users will
stop using the application due to poor
performance
Two Different Thresholds

3.  Compare network latency across sites
2.  Prove the value of a server upgrade1.  Document the results of QoS changes
Validating Changes

Solving Problems
Pervasiveness:
The problem is
effecting user
across your
network

Troubleshooting VoIP

Don’t Commit a Felony

Using Indices
•  Network Congestion Index
•  Packet Loss SLAs
NCI = (Packets/sec + Avg Payload) * (Avg Latency + Avg Bandwidth)
App Owner Controlled Network Controlled
bps < min(rwin/rtt, MSS/(rtt*sqrt(loss)))
For example, to achieve a gigabit per second with TCP on a coast-to-coast
path (rtt = 40 msec), with 1500 byte packets, the loss rate can not exceed
8.5x10^-8! If the loss rate was even 0.1% (far better than most SLAs), TCP
would be limited to just over 9 Mbps. [Note that large packet sizes help. If
packets were n times larger, the same throughput could be achieved with n^2
times as much packet loss.]

Let’s keep the
conversation going…
Andrew.P.White@Gmail.com
ReverendDrew
SystemsManagementZen.Wordpress.com
systemsmanagementzen.wordpress.com/feed/
@SystemsMgmtZen
ReverendDrew
APWhite@us.ibm.com
614-306-3434

Brighttalk learning to cook- network management recipes - final

Brighttalk learning to cook- network management recipes - final

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Brighttalk learning to cook- network management recipes - final

Similar to Brighttalk learning to cook- network management recipes - final (20)

Recently uploaded

Recently uploaded (20)

Brighttalk learning to cook- network management recipes - final