4. PoP Topologies
ī° Core routers â high speed trunk
connections
ī° Distribution routers and Access routers â
high port density
ī° Border routers â connections to other
providers
ī° Service routers â hosting and servers
ī° Some functions might be handled by a
single router
4
5. PoP Design
ī° Modular Design
ī° Aggregation Services separated according
to
īŽ connection speed
īŽ customer service
īŽ contention ratio
īŽ security considerations
5
6. Modular PoP Design
6
Backbone link
to another PoP
Backbone link
to another PoP
Leased line customer
aggregation layer
for leased line circuit delivery
Channelised circuits
Network
Operations
Centre
Consumer
DIal Access
Network
Core
Consumer cable,
xDSL and
wireless Access
for MetroE circuit delivery
GigE fibre trunks
MetroE customer
aggregation layer
ISP Services
(DNS, Mail, News,
FTP, WWW)
Hosted Services &
Datacentre
Other ISPs
Web Cache
7. Modular Routing Protocol Design
Smaller ISPs
ī° Modular IGP implementation
īŽ IGP âareaâ per PoP
īŽ Core routers in backbone area (Area 0/L2)
īŽ Aggregation/summarisation where possible
into the core
ī° Modular iBGP implementation
īŽ BGP route reflector cluster per module
īŽ Core routers are the route-reflectors
īŽ Remaining routers are clients & peer with
route-reflectors only
7
8. Modular Routing Protocol Design
Larger ISPs
ī° Modular IGP implementation
īŽ IGP âareaâ per module (but avoid overloading
core routers)
īŽ Core routers in backbone area (Area 0/L2)
īŽ Aggregation/summarisation where possible
into the core
ī° Modular iBGP implementation
īŽ BGP route reflector cluster per module
īŽ Dedicated route-reflectors adjacent to core
routers
īŽ Clients peer with route-reflectors only
8
10. PoP Modules
ī° Low Speed customer connections
īŽ PSTN/ISDN dialup
īŽ Low bandwidth needs
īŽ Low revenue, large numbers
ī° Leased line customer connections
īŽ E1/T1 speed range
īŽ Delivery over channelised media
īŽ Medium bandwidth needs
īŽ Medium revenue, medium numbers
10
11. PoP Modules
ī° Broad Band customer connections
īŽ xDSL, Cable and Wireless
īŽ High bandwidth needs
īŽ Low revenue, large numbers
ī° MetroE & Highband customer connections
īŽ Trunk onto GigE or 10GigE of 10Mbps and
higher
īŽ Channelised OC3/12 delivery of E3/T3 and
higher
īŽ High bandwidth needs
īŽ High revenue, low numbers
11
12. PoP Modules
ī° PoP Core
īŽ Two dedicated routers
īŽ High Speed interconnect
īŽ Backbone Links ONLY
īŽ Do not touch them!
ī° Border Network
īŽ Dedicated border router to other ISPs
īŽ The ISPâs âfrontâ door
īŽ Transparent web caching?
īŽ Two in backbone is minimum guarantee for
redundancy 12
14. PoP Modules
ī° Network Operations Centre
īŽ Consider primary and backup locations
īŽ Network monitoring
īŽ Statistics and log gathering
īŽ Direct but secure access
ī° Out of Band Management Network
īŽ The ISP Network âSafety Beltâ
14
15. Low Speed Access Module
15
To Core Routers
Primary Rate T1/E1
PSTN lines to
modem bank
PSTN lines to
built-in modems
AS5400
2811
2800/3800
TACACS+/Radius
proxy, DNS resolver,
Content
Web Cache
Access Network
Gateway Routers
16. Medium Speed Access Module
16
To Core Routers
Channelised T1/E1
64K and nx64K circuits
Mixture of channelised
T1/E1, 56/64K and
nx64K circuits
3800/7206/7600
17. High Speed Access Module
17
To Core Routers
Metro Ethernet
Channelised T3/E3
Channelised OC3/OC12
7200/7600/ASR1000/ASR9000
18. Broad Band Access Module
18
To Core Routers
Telephone Network
The cable system
6400
SSG, DHCP, TACACS+
or Radius Servers/Proxies,
DNS resolver, Content
Web Cache
Access Network
Gateway Routers
uBR7246
61xx
IP, ATM
21. Border Module
21
To core routers
Network
Border Routers
To local IXP -
NB - no default route +
local AS routing table only
ISP1 ISP2
22. NOC Module
22
Primary DNS
To core routers
Hosted Network
Gateway Routers
SYSLOG
server
TACACS+
server
Network Operations Centre Staff
Out of Band
Management Network
2811/32async
NetFlow
Analyser
Firewall
Billing, Database
and Accounting
Systems
Corporate LAN
Critical Services
Module
23. Out of Band Network
23
Out of Band
Management Network
2811/32async
To the NOC
Out of Band Ethernet
NetFlow
Collector
NetFlow
enabled
routers
Router
consoles
25. Backbone Design
ī° Routed Backbone
ī° Switched Backbone
īŽ Virtually obsolete
ī° Point-to-point circuits
īŽ nx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48,
10GigE, OC192, OC768
ī° ATM/Frame Relay service from telco
īŽ T3, OC3, OC12,âĻ delivery
īŽ Easily upgradeable bandwidth (CIR)
īŽ Almost vanished in availability now
25
26. Distributed Network Design
ī° PoP design âstandardisedâ
īŽ operational scalability and simplicity
ī° ISP essential services distributed around
backbone
ī° NOC and âbackupâ NOC
ī° Redundant backbone links
26
27. Distributed Network Design
27
POP One
POP Two
POP Three
Customer
connections
Customer
connections
Customer
connections
External
connections
External
connections Operations Centre
Backup
Operations Centre
ISP Services
ISP Services
ISP Services
28. Backbone Links
ī° ATM/Frame Relay
īŽ Virtually disappeared due to overhead, extra
equipment, and shared with other customers
of the telco
īŽ MPLS has replaced ATM & FR as the telco
favourite
ī° Leased Line/Circuit
īŽ Most popular with backbone providers
īŽ IP over Optics and Metro Ethernet very
common in many parts of the world
28
29. Long Distance Backbone Links
ī° Tend to cost more
ī° Plan for the future (at least two years
ahead) but stay in budget
īŽ Unplanned âemergencyâ upgrades can be
disruptive without redundancy
ī° Allow sufficient capacity on alternative
paths for failure situations
īŽ Sufficient can be 20% to 50%
īŽ Some businesses choose 0% â meaning they
have no spare capacity at all!!
29
31. Metropolitan Area Backbone Links
ī° Tend to be cheaper
īŽ Circuit concentration
īŽ Choose from multiple suppliers
ī° Think big
īŽ More redundancy
īŽ Less impact of upgrades
īŽ Less impact of failures
31
32. Metropolitan Area Backbone Links
32
POP One
POP Two
POP Three
Metropolitan Links
Metropolitan Links
Traditional Point to Point Links
34. ISP Services
ī° Most ISP services such as DNS, Mail, etc are
easily deliverable on low budget hardware
platforms
īŽ Single Rack Unit in height (1RU)
īŽ Dual processor is âdefaultâ now
īŽ RAM is very cheap (may as well use 2Gbytes or more)
īŽ Hard drives are very cheap (SCSI more reliable)
īŽ Unix like operating systems (FreeBSD, Debian, Ubuntu,
CentOS) are very common
ī° In addition to commercial operating systems such as
Solaris, RedHat Enterprise Linux &c
īŽ Minimal overhead, minimal OS install, plus the service
required
34
35. ISP Services:
DNS
ī° Domain Name System
īŽ Provides name and address resolution
īŽ Servers need to be differentiated, properly
located and specified
ī° Primary nameserver
ī° Secondary nameserver
ī° Caching nameserver â resolver
35
36. ISP Services:
DNS
ī° Primary nameserver
īŽ Holds ISP zone files
ī° Forward zone (list of name to address mappings) for
all ISPâs and any customer zones
ī° Reverse zone (list of address to name mappings) for
all ISPâs address space
īŽ Hardware & OS: easily satisfied by simple
specification
īŽ Located in secure part of net, e.g. NOC LAN
īŽ Usually run as âhidden masterâ â secondary
nameservers are the official listed nameservers
36
37. ISP Services:
DNS
ī° Secondary nameserver
īŽ Holds copies of ISP zone files
īŽ At least two are required, more is better
īŽ Hardware & OS: easily satisfied by simple
specification
īŽ Strongly recommended to be geographically
separate from each other and the primary DNS
ī° At different PoPs
ī° On a different continent e.g. via services offered by
ISC, PCH and others
ī° At another ISP
37
38. ISP Services:
Secondary DNS Example
38
$ dig apnic.net ns
;; ANSWER SECTION:
apnic.net. 10800 NS ns1.apnic.net.
apnic.net. 10800 NS ns3.apnic.net.
apnic.net. 10800 NS ns4.apnic.net.
apnic.net. 10800 NS ns5.apnic.com.
apnic.net. 10800 NS cumin.apnic.net.
apnic.net. 10800 NS ns-sec.ripe.net.
apnic.net. 10800 NS tinnie.arin.net.
apnic.net. 10800 NS tinnie.apnic.net.
;; ADDITIONAL SECTION:
ns1.apnic.net. 3600 A 202.12.29.25
ns3.apnic.net. 3600 A 202.12.28.131
ns4.apnic.net. 3600 A 202.12.31.140
ns5.apnic.com. 10800 A 203.119.43.200
cumin.apnic.net. 3600 A 202.12.29.59
tinnie.apnic.net. 3600 A 202.12.29.60
ns-sec.ripe.net. 113685 A 193.0.0.196
tinnie.arin.net. 10800 A 199.212.0.53
Tokyo
Hong Kong
Washington
Brisbane
Brisbane
Amsterdam
Washington
39. ISP Services:
Secondary DNS Example
ī° apnic.net zone
īŽ Primary DNS in Brisbane (ns1.apnic.net)
īŽ Secondary DNS run all over the world by APNIC:
ī° Brisbane
ī° Hong Kong
ī° Tokyo
ī° Washington
īŽ Zone secondaried by
ī° RIPE NCC in Amsterdam
ī° ARIN in Washington
īŽ Geographical and service provider redundancy â this is
the perfect example!
39
40. ISP Services:
DNS
ī° Caching nameserver
īŽ This is the resolver â it is the DNS cache
īŽ Your customers use this as resolver, NEVER
your primary or secondary DNS
īŽ Provides very fast lookups
īŽ Does NOT secondary any zones
īŽ One, or preferably two per PoP (redundancy)
īŽ Hardware & OS: easily satisfied by simple
specification
40
41. ISP Services:
Caching Nameserver
ī° DIAL users automatically given the IP
addresses of DNS caches when they dial in 41
To Core Routers
DIAL network
Web Cache
DNS Cache DNS Cache
Radius proxy
Switch redundancy
Router redundancy
DNS Cache redundancy
42. ISP Services:
Anycasting the Caching Nameserver
ī° One trick of the trade
īŽ assign two unique IP addresses to be
used for the two DNS resolver systems
īŽ use these two IP addresses in every PoP
īŽ route the two /32s across your backbone
īŽ even if the two resolver systems in the local
PoP are down, the IGP will ensure that the next
nearest resolvers will be reachable
īŽ Known as IP Anycast
42
Geek
Alert
43. ISP Services:
DNS
ī° Efficient and resilient design
īŽ Primary DNS â keep it secure
īŽ Secondary DNS â geographical and provider
redundancy
ī° Donât ever put them on the same LAN, switched or
otherwise
ī° Donât put them in the same PoP
īŽ Caching DNS â one or two per PoP
īŽ Reduces DNS traffic across backbone
īŽ More efficient, spreads the load
43
44. ISP Services:
DNS
ī° Software
īŽ Make sure that the BIND distribution on the
Unix system is up to date
ī° The vendorâs distribution is rarely current
īŽ Pay attention to bug reports, security issues
īŽ Reboot the DNS cache on a regular (e.g.
monthly) basis
ī° Clears out the cache
ī° Releases any lost RAM
ī° Accepted good practice by system administrators
44
45. ISP Services:
DNS
ī° Implementation
īŽ Put all your hosts, point-to-point links and
loopbacks into the DNS
ī° Under your ISPâs domain name
ī° Use sensible/meaningful names
īŽ Put all your hosts, point-to-point links and
loopbacks into the REVERSE DNS also
ī° Donât forget about in-addr.arpa and ip6.arpa â many
ISPs do
ī° Some systems demand forward/reverse DNS
mapping before allowing access
45
46. ISP Services:
Mail
ī° Must have at least two mail hosts (MX records)
for all supported domains
īŽ Geographical separation helps
ī° Dedicated POP3 server
īŽ Consumers/mobile users get mail from here
ī° SMTP gateway dedicated to that function
īŽ Consumers/mobile users send mail via here
ī° Mail relay open to CUSTOMERS only!
īŽ Donât let outside world use your mail relay
ī° Block port 25 outbound for all customers
īŽ Insist that outbound e-mail goes through SMTP relay
īŽ SMTP relay does virus (ClamAV) and spam
(Spamassassin) filtering
46
47. ISP Services:
Mail Configuration
47
smtp.isp.net
Customer mail relay
Incoming mail
from customer
mail.isp.net
ISP Mail Gateway
Incoming mail
from Internet
pop3.isp.net
Customer POP3/IMAP server
Mail pulled by
customer client
Mail out to
the Internet
SpamAssassin
ClamAV
SpamAssassin
ClamAV
48. ISP Services:
Mail Example
ī° cisco.com mail (MX records)
īŽ primary MX are 6 systems in San Jose
īŽ Three backup MXes in RTP, Amsterdam and Sydney
īŽ backup MX only used if primary unavailable
48
$ dig cisco.com mx
;; ANSWER SECTION:
cisco.com. 86400 MX 10 sj-inbound-a.cisco.com.
cisco.com. 86400 MX 10 sj-inbound-b.cisco.com.
cisco.com. 86400 MX 10 sj-inbound-c.cisco.com.
cisco.com. 86400 MX 10 sj-inbound-d.cisco.com.
cisco.com. 86400 MX 10 sj-inbound-e.cisco.com.
cisco.com. 86400 MX 10 sj-inbound-f.cisco.com.
cisco.com. 86400 MX 15 rtp-mx-01.cisco.com.
cisco.com. 86400 MX 20 ams-inbound-a.cisco.com.
cisco.com. 86400 MX 25 syd-inbound-a.cisco.com.
49. ISP Services:
Mail
ī° Software
īŽ Make sure that the MAIL and POP3
distributions on the Unix system are up to date
ī° The vendor distributions are rarely current
īŽ Pay attention to bug reports, security issues,
unsolicited junk mail complaints
49
IMPORTANT: Do NOT allow non-customers
to use your mail system as a relay
50. ISP Services:
News
ī° News servers provide a Usenet news feed
to customers
ī° Distributed design required
īŽ Incoming newsfeed to one large server
īŽ Distributed to feed servers in each PoP
īŽ Feed servers provide news feed to customers
īŽ Outgoing news goes to another server
īŽ Separate reading news system
īŽ Separate posting news system
50
51. ISP Services:
News System Placement
51
POP One
POP Two
POP Three
Customer
connections
Customer
connections
Customer
connections
External
connections
External
connections News Collector
News Feeder
News Feeder
News Feeder
News Distributor
52. ISP Services:
News System Placement
52
POP One
POP Two
POP Three
Customer
connections
Customer
connections
Customer
connections
External
connections
External
connections News Collector
News Feeder
News Feeder
News Feeder
News Distributor
53. ISP Services:
News
ī° Software
īŽ Make sure that the Internet News distribution
on the Unix system is up to date
ī° The vendor distributions are rarely current
īŽ Pay attention to bug reports, security issues,
unsolicited junk posting complaints
53
IMPORTANT: Do NOT allow non-customers
to use your news system for posting messages
55. Where to get IP addresses and AS
numbers
ī° Your upstream ISP
ī° Africa
īŽ AfriNIC â http://www.afrinic.net
ī° Asia and the Pacific
īŽ APNIC â http://www.apnic.net
ī° North America
īŽ ARIN â http://www.arin.net
ī° Latin America and the Caribbean
īŽ LACNIC â http://www.lacnic.net
ī° Europe and Middle East
īŽ RIPE NCC â http://www.ripe.net/info/ncc
55
57. Getting IP address space
ī° Take part of upstream ISPâs PA space
or
ī° Become a member of your Regional Internet
Registry and get your own allocation
īŽ Require a plan for a year ahead
īŽ General policies are outlined in RFC2050, more
specific details are on the individual RIR website
ī° There is no more IPv4 address space at IANA
īŽ Most RIRs are now entering their âfinal /8â IPv4
delegation policies
īŽ Limited IPv4 available
īŽ IPv6 allocations are simple to get in most RIR regions
57
58. What about RFC1918 addressing?
ī° RFC1918 defines IP addresses reserved for
private Internets
īŽ Not to be used on Internet backbones
īŽ http://www.ietf.org/rfc/rfc1918.txt
ī° Commonly used within end-user networks
īŽ NAT used to translate from private internal to public
external addressing
īŽ Allows the end-user network to migrate ISPs without a
major internal renumbering exercise
ī° Most ISPs filter RFC1918 addressing at their
network edge
īŽ http://www.cymru.com/Documents/bogon-
list.html 58
59. What about RFC1918 addressing?
ī° List of well known problems with this approach for
an SP backbone:
īŽ Breaks Path MTU Discovery
īŽ Potential conflicts with usage of private addressing inside
customer networks
īŽ Security through obscurity does not provide security
īŽ Troubleshooting outside the local network becomes very
hard
ī° Router interface addresses are only locally visible
ī° Internet becomes invisible from the router
īŽ Troubleshooting of connectivity issues on an Internet scale
becomes impossible
ī° Traceroutes and pings provide no information
ī° No distinction between ânetwork invisibleâ and ânetwork
brokenâ
īŽ Increases operational complexity of the network
infrastructure and routing configuration
59
60. Private versus Globally Routable IP
Addressing
ī° Infrastructure Security: not improved by using
private addressing
īŽ Still can be attacked from inside, or from customers, or
by reflection techniques from the outside
ī° Troubleshooting: made an order of magnitude
harder
īŽ No Internet view from routers
īŽ Other ISPs cannot distinguish between down and broken
ī° Performance: PMTUD breakage
ī° Summary:
īŽ ALWAYS use globally routable IP addressing for ISP
Infrastructure
60
61. Addressing Plans â ISP
Infrastructure
ī° Address block for router loop-back interfaces
ī° Address block for infrastructure
īŽ Per PoP or whole backbone
īŽ Summarise between sites if it makes sense
īŽ Allocate according to genuine requirements, not historic
classful boundaries
ī° Similar allocation policies should be used for IPv6
as well
īŽ ISPs just get a substantially larger block (relatively) so
assignments within the backbone are easier to make
61
62. Addressing Plans â Customer
ī° Customers are assigned address space
according to need
ī° Should not be reserved or assigned on a
per PoP basis
īŽ ISP iBGP carries customer nets
īŽ Aggregation not required and usually not
desirable
62
63. Addressing Plans â ISP Infrastructure
ī° Phase One
63
223.10.0.0/21
Customer assignments Infrastructure Loopbacks
/24
223.10.6.255
223.10.0.1
223.10.0.0/20
Original assignments New Assignments
/24
/24
223.10.0.1
223.10.5.255 223.10.15.255
Phase Two
64. Addressing Plans
Planning
ī° Registries will usually allocate the next
block to be contiguous with the first
allocation
īŽ Minimum allocation could be /21
īŽ Very likely that subsequent allocation will
make this up to a /20
īŽ So plan accordingly
64
65. Addressing Plans (contd)
ī° Document infrastructure allocation
īŽ Eases operation, debugging and management
ī° Document customer allocation
īŽ Contained in iBGP
īŽ Eases operation, debugging and management
īŽ Submit network object to RIR Database
65
67. Routing Protocols
ī° IGP â Interior Gateway Protocol
īŽ carries infrastructure addresses, point-to-point
links
īŽ examples are OSPF, ISIS,...
ī° EGP â Exterior Gateway Protocol
īŽ carries customer prefixes and Internet routes
īŽ current EGP is BGP version 4
ī° No connection between IGP and EGP
67
68. Why Do We Need an IGP?
ī° ISP backbone scaling
īŽ Hierarchy
īŽ Modular infrastructure construction
īŽ Limiting scope of failure
īŽ Healing of infrastructure faults using dynamic
routing with fast convergence
68
69. Why Do We Need an EGP?
ī° Scaling to large network
īŽ Hierarchy
īŽ Limit scope of failure
ī° Policy
īŽ Control reachability to prefixes
īŽ Merge separate organizations
īŽ Connect multiple IGPs
69
70. Interior versus Exterior Routing
Protocols
ī° Interior
īŽ Automatic neighbour
discovery
īŽ Generally trust your IGP
routers
īŽ Prefixes go to all IGP
routers
īŽ Binds routers in one AS
together
ī° Exterior
īŽ Specifically configured
peers
īŽ Connecting with outside
networks
īŽ Set administrative
boundaries
īŽ Binds ASâs together
70
71. Interior versus Exterior Routing
Protocols
ī° Interior
īŽ Carries ISP
infrastructure addresses
only
īŽ ISPs aim to keep the
IGP small for efficiency
and scalability
ī° Exterior
īŽ Carries customer
prefixes
īŽ Carries Internet
prefixes
īŽ EGPs are independent
of ISP network topology
71
72. Hierarchy of Routing Protocols
72
BGP4
BGP4
and OSPF/ISIS
Other ISPs
Customers
IXP
Static/BGP4
BGP4
73. Routing Protocols:
Choosing an IGP
ī° Review the âOSPF vs ISISâ presentation:
īŽ OSPF and ISIS have very similar properties
ī° ISP usually chooses between OSPF and
ISIS
īŽ Choose which is appropriate for your operatorsâ
experience
īŽ In most vendor releases, both OSPF and ISIS
have sufficient ânerd knobsâ to tweak the IGPâs
behaviour
īŽ OSPF runs on IP
īŽ ISIS runs on infrastructure, alongside IP
73
74. Routing Protocols:
IGP Recommendations
ī° Keep the IGP routing table as small as possible
īŽ If you can count the routers and the point to point links
in the backbone, that total is the number of IGP entries
you should see
ī° IGP details:
īŽ Should only have router loopbacks, backbone WAN
point-to-point link addresses, and network addresses of
any LANs having an IGP running on them
īŽ Strongly recommended to use inter-router
authentication
īŽ Use inter-area summarisation if possible
74
75. Routing Protocols:
More IGP recommendations
ī° To fine tune IGP table size more, consider:
īŽ Using âip unnumberedâ on customer point-to-
point links â saves carrying that /30 in IGP
ī° (If customer point-to-point /30 is required for
monitoring purposes, then put this in iBGP)
īŽ Use contiguous addresses for backbone WAN
links in each area â then summarise into
backbone area
īŽ Donât summarise router loopback addresses â
as iBGP needs those (for next-hop)
īŽ Use iBGP for carrying anything which does not
contribute to the IGP Routing process
75
76. Routing Protocols:
iBGP Recommendations
ī° iBGP should carry everything which
doesnât contribute to the IGP routing
process
īŽ Internet routing table
īŽ Customer assigned addresses
īŽ Customer point-to-point links
īŽ Dial network pools, passive LANs, etc
76
77. Routing Protocols:
More iBGP Recommendations
ī° Scalable iBGP features:
īŽ Use neighbour authentication
īŽ Use peer-groups to speed update process and
for configuration efficiency
īŽ Use communities for ease of filtering
īŽ Use route-reflector hierarchy
ī° Route reflector pair per PoP (overlaid clusters)
77
79. Security
ī° ISP Infrastructure security
ī° ISP Network security
ī° Security is not optional!
ī° ISPs need to:
īŽ Protect themselves
īŽ Help protect their customers from the Internet
īŽ Protect the Internet from their customers
ī° The following slides are general recommendations
īŽ Do more research on security before deploying any
network
79
80. ISP Infrastructure Security
ī° Router security
īŽ Usernames, passwords, vty filters, TACACS+
īŽ Disable telnet on vtys, only use SSH
īŽ vty filters should only allow NOC access, no
external access
īŽ See IOS Essentials for the recommended
practices for ISPs
80
81. ISP Infrastructure Security
ī° ISP server security
īŽ Usernames, passwords, TCP wrappers,
IPTABLES
īŽ Protect all servers using routers with strong
filters applied
ī° Hosted services security
īŽ Protect network from hosted servers using
routers with strong filters
īŽ Protect hosted servers from Internet using
routers with strong filters
81
82. ISP Infrastructure Security
ISP Server Protection
82
DNS
cache
DNS
secondary
POP3
Mail
Relay
NEWS
To core routers
Service Network
Gateway Routers
Access-list examples:
Allow tcp/established to all servers
ICMP
DNS 2ary: udp/53 and tcp/53
POP3: tcp/110
Mail Relay: tcp/25 and ISP address
range only
News: tcp/119 and ISP
address range only
DNS Cache: udp/53
Web server: tcp/80
Other necessary filters:
All servers: SSH (tcp/22) from NOC LAN only
Web
server
83. ISP Infrastructure Security
Hosted Server Protection
83
Access-list examples:
Inbound
Allow tcp/established to all servers
ICMP
Web server: tcp/80
SSH for customer access
Any other ports for services
sold to customers
Outbound
ICMP
Allow DNS udp/53 and
tcp/53
Block all access to ISP
address range
Server5
Server1 Server2 Server3 Server4
To core routers
Service Network
Gateway Routers
Server6
85. ISP Network Security
ī° Denial of Service Attacks
īŽ eg: âsmurfingâ
īŽ see http://www.denialinfo.com
ī° Effective filtering
īŽ Network borders â see Cisco ISP Essentials
īŽ Customer connections â unicast RPF on ALL of
them
īŽ Network operation centre
īŽ ISP corporate network â behind firewall
85
86. ISP Network Security
Secure external access
ī° How to provide staff access from outside
īŽ Set up ssh gateway (Unix system with ssh
daemon and nothing else configured)
īŽ Provide ssh client on all staff laptops
īŽ ssh available on Unix and Windows
īŽ ssh is Secure Shell â encrypted link
ī° How not to provide access from outside
īŽ telnet, rsh, rlogin â these are all insecure
īŽ Open host â insecure, can be compromised
86
87. Ingress & Egress Route Filtering
Your customers should not
be sending any IP packets
out to the Internet with a
source address other then
the address you have
allocated to them!
87
89. Out of Band Management
ī° Not optional!
ī° Allows access to network equipment in
times of failure
ī° Ensures quality of service to customers
īŽ Minimises downtime
īŽ Minimises repair time
īŽ Eases diagnostics and debugging
89
90. Out of Band Management
ī° OoB Example â Access server:
īŽ modem attached to allow NOC dial in
īŽ console ports of all network equipment
connected to serial ports
īŽ LAN and/or WAN link connects to network
core, or via separate management link to NOC
ī° Full remote control access under all
circumstances
90
91. Out of Band Network
91
Ethernet
to the NOC
Router, switch
and ISP server
consoles
(Optional) Out of band
WAN link to other PoPs
Modem â access
to PSTN for out of
band dialin
Equipment Rack
Equipment Rack
92. Out of Band Management
ī° OoB Example â Statistics gathering:
īŽ Routers are NetFlow and syslog enabled
īŽ Management data is congestion/failure
sensitive
īŽ Ensures management data integrity in case of
failure
ī° Full remote information under all
circumstances
92
94. Test Laboratory
ī° Designed to look like a typical PoP
īŽ Operated like a typical PoP
ī° Used to trial new services or new software
under realistic conditions
ī° Allows discovery and fixing of potential
problems before they are introduced to
the network
94
95. Test Laboratory
ī° Some ISPs dedicate equipment to the lab
ī° Other ISPs âpurchase aheadâ so that
todayâs lab equipment becomes
tomorrowâs PoP equipment
ī° Other ISPs use lab equipment for âhot
sparesâ in the event of hardware failure
95
96. Test Laboratory
ī° Canât afford a test lab?
īŽ Set aside one spare router and server to trial
new services
īŽ Never ever try out new hardware, software or
services on the live network
ī° Every major ISP in the US and Europe has
a test lab
īŽ Itâs a serious consideration
96
99. Operational Considerations
Maintenance
ī° Never work on the live network, no matter
how trivial the modification may seem
īŽ Establish maintenance periods which your
customers are aware of
ī° e.g. Tuesday 4-7am, Thursday 4-7am
ī° Never do maintenance on a Friday
īŽ Unless you want to work all weekend cleaning
up
ī° Never do maintenance on a Monday
īŽ Unless you want to work all weekend preparing
99
100. Operational Considerations
Support
ī° Differentiate between customer support
and the Network Operations Centre
īŽ Customer support fixes customer problems
īŽ NOC deals with and fixes backbone and
Internet related problems
ī° Network Engineering team is last resort
īŽ They design the next generation network,
improve the routing design, implement new
services, etc
īŽ They do not and should not be doing support!
100
101. Operational Considerations
NOC Communications
ī° NOC should know contact details for
equivalent NOCs in upstream providers
and peers
ī° Or consider joining the INOC-DBA system
īŽ Voice over IP phone system using SIP
īŽ Runs over the Internet
īŽ www.pch.net/inoc-dba for more information
101
103. ISP Design Summary
ī° KEEP IT SIMPLE & STUPID ! (KISS)
ī° Simple is elegant is scalable
ī° Use Redundancy, Security, and
Technology to make life easier for yourself
ī° Above all, ensure quality of service for
your customers
103