Implementing Internet and MPLS BGP

Implementing & Troubleshooting
BGP
Tanner
5/23/2008
5/30/2008

Agenda
PART 1
BGP Fundamentals
BGP and the Internet
PART 2
BGP and the WAN
Troubleshooting
2

PART 1
BGP Fundamentals
BGP and the Internet
3

BGP Fundamentals
Operations
4

Where is BGP used?
Internet
Same, Unique, or Mixed ASN
MPLS WAN
Public or Private ASN
5

ASAutonomous Systems
Textbook answer:
An AS is a connected group of one or more IP prefixes run by one or more network operators which has a SINGLE and CLEARLY DEFINED routing policy.
6

ASN’sAutonomous System Numbers
16-bit ASN’s (RFC1930)
Range: 0-65535
Public: 1-64511
Private: 64512-65534
32-bit ASN’s (RFC4893)
4-octets
0.0 to 65535.65535
Only 46 32-bit ASN’s currently allocated
7

EIGRP and BGP Comparison
8

Path Selection
Attributes
Highest Weight
Highest Local Preference
Internally Originated
Shortest AS-Path
Manipulating these attributes changes BGP path selection
9

Terminology
attribute [noun]
Pronunciation: a-trə-byüt
3: a word ascribing a quality; especially
attribute [transitive verb]
Pronunciation: ə-tri-byüt, -byət
1: to explain by indicating a cause <attributed his success to his coach>
10

BGP AttributesRFC1771 attributes its success to its attributes
11

BGP AttributesMost Used
Influence INBOUNDTraffic
The transit path to you is determined by how you announce your routes
AS Path Prepend (shorter is more preferred)
route-map RM-ISP-OUT
set as-path prepend123 123 123
Influence OUTBOUNDtraffic
Local Preference (higher is more preferred)
route-map RM-ISP-IN
set local-preference 50
12

BGP Process Operations
13

Section ReviewFundamentals
What is an autonomous system?
What are BGP attributes that affect inbound traffic?
What are BGP attributes that affect outbound traffic?
Name 4 common BGP path selection criteria
What maintenance task happens every 60 seconds in BGP?
14

BGP & the Internet
15
23nd Ave / I-40 Junction

Global IP Assignments
IANA
Regional Registrars
ISP’s
End Users
16

Address Space DepletionBGP Movie (6 min)
17

Global Routing tableHow large is it?
.:: Limit Prefixes on Cisco routers
router bgp 12345
neighbor 1.1.1.1 maximum-prefix 300000 90
18

RIR whois
ARIN IP Lookup
AfriNIC Country Lookup
19

Typical ISP Routing Options
Single-homed, Single ISP
Private AS or Static (No BGP)
Multi-homed, Single ISP
Private AS
Multi-homed, Dual ISP
Public AS
20

Prefix OriginationInbound Traffic
Common Elements
ISP’s won’t accept anything longer than /24
Provider Aggregate address block (PA)
/24 or shorter from ISP
Justification paperwork, but usually easy
Announcing another ISP’s prefix
Provider Independent address block (PI)
Applied for from RIR (e.g., ARIN)
More Paperwork (and solid justification)!
21

What Kind of Routes?Outbound Traffic
1Based on 2 upstream eBGP peers
2 Varies depending on quantity of ISP customers announcing prefixes
3Varies depending on size of upstream carrier
4 Inbound bogon filtering is still possible, however outbound will not function due to default route
22

Memory RequirementsFull BGP Routes
Based on 255K routes + soft reconfig
BGP Summary shows 57MB Used
BGP-Router# shipbgp sum
...
BGP using 57060899 total bytes of memory
Adding up processes shows 153MB Used
BGP-Router# sh proc mem | i PID|BGP
PID TTY Allocated Freed Holding GetbufsRetbufs Process
215 0 152845892 1430904 145443600 16 16 BGP Router
234 0 239016 0 6984 5164371 5164371 BGP I/O
235 0 0 82472 9972 0 0 BGP Scanner
23

BGP Policy Components
Prefix-lists to filter prefixes
ip prefix-list PL-ANNOUNCE seq 10 permit 1.0.0.0/8
Filter-lists to filter ASNs
ip as-path access-list 1 ^1234
Route-maps to apply policy
route-map RM-ISP-OUT permit 10 set as-path prepend 1234
Distribute-lists to sit and watch (don’t use)
Source: NANOG 23, Phillip Smith
24

Configuration ExampleISP eBGP Peer with Partial Routes
router bgp 1234
no auto-summary
no synchronization
no bgp fast-external-fallover
bgp log-neighbor-changes
neighbor 192.0.2.233 remote-as 209
neighbor 192.0.2.233 description eBGP with Qwest AS209. Password: 1234abcd
neighbor 192.0.2.233 password 1234abcd
neighbor 192.0.2.233 version 4
neighbor 192.0.2.233 soft-reconfiguration inbound
neighbor 192.0.2.233 maximum-prefix 300000 90 warning-only
neighbor 192.0.2.233 prefix-list PL-BOGONS in
neighbor 192.0.2.233 prefix-list PL-ANNOUNCE out
neighbor 192.0.2.233 route-map RM-QWEST-OUT out
neighbor 192.0.2.233 route-map RM-QWEST-IN in
neighbor 192.0.2.233 filter-list 1 out
neighbor 192.0.2.233 filter-list 10 in
network 205.93.251.0 mask 255.255.254.0
network 205.93.251.0
ip route 205.93.251.0 255.255.254.0 Null0 name BGP-STABILITY
ip route 205.93.251.0 255.255.255.0 205.93.251.4
ip route 205.93.251.125 255.255.255.255 205.93.251.121 name IBGP-PEER
ip route 205.93.251.125 255.255.255.255 205.93.251.2 250 name IBGP-PEER-BACKUP
ip as-path access-list 1 permit ^$
ip as-path access-list 10 permit _(209|7018)$
ip prefix-list PL-ANNOUNCE seq 10 permit 205.93.251.0/23 le 24
ip prefix-list PL-ANNOUNCE seq 99 deny 0.0.0.0/0 le 32
route-map RM-QWEST-OUT permit 10
set as-path prepend 1234 1234
route-map RM-QWEST-IN permit 10
set local-pref 50
25

Regular ExpressionsBGP AS Filtering
Defining our AS
ip as-path access-list 1 permit ^$
Originating in AS 3549
ip as-path access-list 1 permit ^3549$
Originating in AS 3549 or Upstream AS
ip as-path access-list 1 permit ^3549$
ip as-path access-list 1 permit ^3549 1239$
ip as-path access-list 1 permit ^3549_(1239)?$
Deny all nets originating from AS 1239 and permit all other routes
ip as-path access-list 1 deny _1239$
ip as-path access-list 1 permit .*
26

BGP Routing Table Analysis
Daily BGP Stats Available
BGP routing table entries examined: 255572
Prefixes after maximum aggregation: 127106
Deaggregation factor: 2.01
Unique aggregates announced to Internet: 123962
Total ASes present in the Internet Routing Table: 28151
Prefixes per ASN: 9.08
Average AS path length visible in the Internet Routing Table: 3.6
Max AS path length visible: 25
Max AS path prepend of ASN (39375) 13
Prefixes from unregistered ASNs in the Routing Table: 25414
Unregistered ASNs in the Routing Table: 1885
Prefixes being announced from unallocated address space: 786
Number of addresses announced to Internet: 1,851,293,088
WojciechMisiaszek
TelekomunikacjaPodlasie Sp.
ul. Dobra 14A
15-034 Bialystok
Poland
27

Bogon FilteringManual Method
28
Outbound traffic (via inbound route filter)
ip prefix-list BOGONS descBogon networks we won't accept
ip prefix-list BOGONS seq 2 deny 0.0.0.0/0
ip prefix-list BOGONS seq 5 deny 0.0.0.0/8 le 32
ip prefix-list BOGONS seq 700 permit 0.0.0.0/0 le 27
Inbound traffic
ip access-list extended ACL-OUTSIDE-IN
remark --- Basic Spoof Filtering
deny ip 0.0.0.0 0.255.255.255 any
deny ip 10.0.0.0 0.255.255.255 any
deny ippublic-ip-blocksubnet-mask any

Bogon FilteringAutomatic Method
Do not try this at home!
Make sure you are aware of potential complications
29
router bgp <your asn>
neighbor x.x.x.x remote-as 65333
neighbor x.x.x.xebgp-multihop 255
neighbor x.x.x.x description CymruBogon Route Server Project
neighbor x.x.x.x prefix-list PL-CYMRU-OUT out
neighbor x.x.x.x route-map RM-CYMRUBOGONS-IN in
neighbor x.x.x.x password 31337PW
neighbor x.x.x.x maximum-prefix 100 threshold 90
Configure a community list to accept the bogon prefixes into the route-map.
ipbgp-community new-format
ip community-list 10 permit 65333:888
Configure the route-map. Remember to apply it to the proper peering sessions.
route-map RM-CYMRUBOGONS-IN permit 10
description Filter bogons learned from cymru.com bogon route-servers
match community 10
set ip next-hop 192.0.2.1
Set a bogon next-hop on all routers that receive the bogons.
ip route 192.0.2.1 255.255.255.255 null0
ip prefix-list PL-CYMRU-OUT seq 5 deny 0.0.0.0/0 le 32

BGP Communities
WELL KNOWN
TE Custom Communities
no-advertise
no-export
ISP must support it
TE via AS path prepends, local prefs, trig. blackhole
30

BGP CommunitiesConfiguration Example
ipbgp-community new-format
ipprefix-list PL-ANNOUNCE seq 10 permit 205.93.251.0/24
ipprefix-list PL-ANNOUNCE seq 10 deny 0.0.0.0/0 le 32
route-map RM-ISP-OUT permit 10
match ip address prefix-list PL-ANNOUNCE
set community 65011:209
route-map RM-ISP-OUT permit 20
router bgp 64512
neighbor 1.1.1.1 send-community
neighbor 1.1.1.1 route-map RM-ISP-OUT out
31

Section ReviewBGP & the Internet
What kind of route options are typically received from an ISP?
Who is the global controller of IP space on the internet?
Describe bogon filtering
What do the ^ and $ symbols mean in regular expressions?
32

PART 2
BGP and the MPLS WAN
Troubleshooting BGP
33

BGP & MPLS
Theory
Design
Configuration
Best Practices
34

MPLS Basics
Topology
Full Mesh
Single peer to WAN cloud
L1 Transport
T1
DS3
L2 Transport
PPP / MLP
ATM / IMA
Frame Relay
Ethernet
Routing Protocols
BGP
EIGRP
RIP
Public/Private AS’s
35

MPLS Terminology
CE Router
Customer Edge
PE Router
Provider Edge
P/LSR Router
Provider Backbone/Label Switching Router
VRF
Virtual Routing and Forwarding
Everything else is standard BGP!
36

Typical MPLS Topology Options
Single-homed, Single ISP
Easiest routing policies
Multi-homed, Single ISP
Most common
Multi-homed, Dual Provider
Lots of TE
37

BGP TableHow do you read this thing???
38
> is the path installed in the routing table
rmeans there is already a route with a better AD
32768means prefix originated on this router
? means prefix was originated via redistribution
Next Hopis the neighbor IP of eBGP peer(s)
WAN-Router# shipbgp
BGP table version is 7345, local router ID is 172.16.254.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* i12.86.42.44/30 172.16.254.4 0 100 0 7018 ?
*> 12.122.14.185 0 7018 ?
r>i172.16.254.4/32 172.16.254.4 0 100 0 ?
*> 172.16.254.16/29 0.0.0.0 0 32768 ?
*> 172.16.254.24/29 0.0.0.0 0 32768 ?
* i172.30.32.0/20 172.16.254.4 0 100 0 7018 7018 i
*> 12.122.14.185 0 7018 7018 i
*> 172.30.64.0/20 12.122.14.185 0 7018 7018 ?
imeans prefix was originated via network statement
AS set is the list of AS’s prefix has passed through
CIDR Mask try to summarize where possible
* means route is OK to inject in routing table
imeans prefix learned from iBGP peer

Default Route Origination
39
*Policies include: Conditional advertisement, AS prepending, and communities

Best Path Selection
Review
BGP Table (BRIB)
Routing Table (RIB)
BGP Multipath
Multi-VRF w/Sub-interfaces
40
Weight
Local Pref
Local Originate
AS Path
Origin Type
Lowest MED
eBGP over iBGP
IGP Metric to NH
Received First
Lowest RID
Originator ID
Neighbor IP
WAN-router# shipbgpnei 172.16.16.249 advertised-routes
Originating default network 0.0.0.0
*> 10.0.0.0/24 10.20.40.5 0 32768 ?
*> 10.20.20.0/24 0.0.0.0 0 32768 ?
...
Only send the very best!
WAN-router> shipbgp
*> 0.0.0.0 172.14.16.250 0 65000 i
* 0.0.0.0 10.217.13.102 0 65001 i
WAN-router> shipbgp
*> 0.0.0.0 172.11.132.193 0 1803 65000 i

Route Redistribution
“Seek first to summarize…”
Do you need to redistribute?
Yes = Redistribution
No = Summarization
Maybe = Both?
BGP to EIGRP
router eigrp 111
redistribute bgp 222 metric 1500 1000 255 1 1500
EIGRP to BGP
router bgp 222
redistribute eigrp 111
41

Miscellaneous Features
Peer Groups
Object-groups for BGP! (Kind of…)
router bgp 64512
! Setup peer-group policies
neighbor PARTIAL-ROUTES peer-group
neighbor PARTIAL-ROUTES version 4
neighbor PARTIAL-ROUTES filter-list 5 out
neighbor FULL-ROUTES peer-group
neighbor FULL-ROUTES version 4
! Apply it to a neighbor
neighbor 192.0.2.228 peer-group FULL-ROUTES
ip as-path access-list 5 permit ^(209|36270|6298_)[0-9]*_[0-9]*$
Route Reflectors
42

Best Practices
Avoid redistributing everything under the sun
connected, static, every routing protocol, etc.
Look for ways to reduce routing tables
Summarize
Advertise only what is necessary
Use a network statement for default origination
network 0.0.0.0 mask 0.0.0.0
43

Case StudyRequirements
WAN to Internet
Use DC as primary
Use Campus as secondary
Use Internet VPN as tertiary
WAN to Hubs
Use each hub MPLS DS3
Use other hub DS3 as secondary
Hub to Hub
Use LAN link as primary
Don’t use MPLS DS3’s as secondary
44
Smokey the Router says…
“Routing works both ways!”

Case StudyPossible Solution
45
WAN to Internet
Use DC as primary
Use Campus as secondary
WAN to Hubs
Use each hub MPLS DS3
Use other hub DS3 as secondary
Hub to Hub
Use LAN link as primary
Don’t use MPLS DS3’s as secondary
Default Networks:
1 via eBGP to MPLS
1 via iBGP to VPN
Advertised Networks:
Shortest AS path (DC)
network 0.0.0.0
network 10.112.0.0
Received Networks:
0.0.0.0/0 ge 29 le 32
network 0.0.0.0
Received Networks:
0.0.0.0/0 ge 29 le 32
Summarized Networks:
summary-addr 10.x.0.0/20
summary-addr <WAN nets>
network 0.0.0.0
Received Networks:
0.0.0.0/0 ge 29 le 32
Summarized Networks:
summary-addr 10.112.0.0/16
Default Route
Static route redistributed into EIGRP
Campus to WAN:
EIGRP Metric better via Router 1  WAN
DC to Campus:
Only 1 route via Interlink

Configuration Example (Hub)MPLS eBGP Peer with Default Advertisement
46
router bgp 100
network 0.0.0.0
network 10.112.0.0 mask 255.255.0.0
neighbor 192.0.2.105 remote-as 65000
neighbor 192.0.2.105 description eBGP with MPLS SP. Password: 1234abcd
neighbor 192.0.2.105 password 1234abcd
neighbor 192.0.2.105 version 4
neighbor 192.0.2.105 send-community
neighbor 192.0.2.105 soft-reconfiguration inbound
neighbor 192.0.2.105 route-map RM-MPLS-IN in
neighbor 192.0.2.105 route-map RM-MPLS-OUT out
no auto-summary
ip prefix-list PL-DEFAULT seq 10 permit 0.0.0.0/0
route-map RM-MPLS-IN deny 10
description Block learning default route from DC Router. Use IGP instead.
match ip address prefix-list PL-DEFAULT
route-map RM-MPLS-IN permit 20
route-map RM-MPLS-OUT permit 10
description Set BGP policies for outbound route advertisements to MPLS Provider
set community 112
description Prepend Default Route for Backup Link
match ip address prefix-list PL-DEFAULT
set as-path prepend 100 100

Configuration Example (Hub)MPLS EIGRP Redistribution
router eigrp 1
redistribute bgp 100 metric 1500 1000 255 1 1500 route-map PL-WAN-SERIALS
network 10.112.2.0 0.0.0.255
no auto-summary
ip prefix-list PL-WAN-SERIALS seq 10 permit 0.0.0.0/0 ge 29
route-map RM-WAN-SERIALS permit 10
description Only redistribute WAN serials (/29 to /32 prefixes) into EIGRP process
match ip address prefix-list PL-WAN-SERIALS
Advertise learned BGP networks with prefixes /29 or longer
47

Section ReviewBGP & MPLS
What are the 3 default route origination methods?
What does the > symbol mean in the BGP table?
What are 3 clues that tell you a route "originated here" in the BGP table?
48

BGP Troubleshooting
Interpreting and Troubleshooting BGP Operations
49

Peer Establishment
Peer Reachability
MD5 Password Mismatch
Wrong neighbor IP
Wrong update-source
Wrong peer AS
TTL / ebgp-multihop
Stuck in OpenSent/OpenConfirm
Asymmetric routing & TTL problem
ACL’s between peers
Blocking TCP/179
50

Flapping Peer
*May 20 04:02:39.240 MST: %BGP-5-ADJCHANGE: neighbor 192.0.2.133 Down Peer closed the session
*May 20 04:02:54.468 MST: %BGP-5-ADJCHANGE: neighbor 192.0.2.133 Up
*May 20 04:20:44.999 MST: %BGP-5-ADJCHANGE: neighbor 192.0.2.133 Down BGP Notification sent
*May 20 04:20:44.999 MST: %BGP-3-NOTIFICATION: sent to neighbor 192.0.2.133 4/0 (hold time expired) 0 bytes
*May 20 04:52:18.132 MST: %BGP-3-NOTIFICATION: sent to neighbor 192.0.2.133 4/0 (hold time expired) 0 bytes
Remote router rebooting (BGP crash?)
MTU Incorrect
L2 Problem
Interface output drops (QoS, CoPP, etc.)
51

Received RoutesPre/Post Filter
Show received routes before policy is applied
shipbgpnei 1.1.1.1 received-routes
Requires soft-reconfiguration inbound (more mem)
Show received routes after policy is applied
shipbgpnei 1.1.1.1 routes
Show AS Paths to via all neighbors
BGP-router> shipbgp paths
Address Hash Refcount Metric Path
0xC4125EDC 1 8 0 7018 209 701 23520 3816 ?
0x68397C58 1 18 0 4323 6389 6198 27266 25747 i
0x74151970 1 2 0 4323 1299 13249 44600 i
0x70FF72D4 1 2 0 4323 3257 1241 20506 i
52

Missing Routes
Next hop IP address must be accessible
iBGP next-hop-self
Route with better AD already exists in RIB
Filters
Prefix
AS-Path
Route-maps
53

AnnouncementsVerify advertised routes
Show advertised routes to peer
shipbgpnei 192.0.2.233 advertised-routes
54
BGP-Router> shipbgpnei 192.0.2.233 advertised-routes
*> 205.93.251.0 205.93.251.4 0 32768 i
*> 205.93.251.0/23 0.0.0.0 0 32768 i
Total number of prefixes 2
What if nothing shows up?
Route must exist in the RIB

BGP TableAnalyzing and Interpreting
Router# shipbgp
*>i0.0.0.0 205.93.251.125 0 100 0 7018 i
*> 3.0.0.0 192.0.2.233 0 4323 1239 701 703 80 i
*> 4.0.0.0/9 192.0.2.233 0 4323 3549 3356 i
* i 205.93.251.125 0 100 0 7018 209 3356 i
*> 4.0.0.0 192.0.2.233 0 4323 3549 3356 i
* i 205.93.251.125 0 100 0 7018 209 3356 i
*> 4.23.112.0/24 192.0.2.233 0 4323 174 21889 i
*>i12.2.60.0/22 205.93.251.125 0 100 0 7018 209 7018 32719 i
* 192.0.2.233 0 4323 6539 19092 26794 26794 26794 26794 26794 26794 26794 26794 26794 26794 26794 32719 i
Note to self: 10 prepends is excessive
Average AS path length is 3.6
55

Looking GlassPublic BGP Route Servers - CLI
Verify how the global internet routing table views your prefix announcement
route-views.oregon-ix.net> shipbgp205.93.251.0 | i64512
3333 3356 1239 4323 64512
2905 701 209 7018 64512
4513 13789 22212 4323 64512
7018 4323 64512
...
56

Looking GlassPublic BGP Route Servers – Web/CLI
57

High CPU
BGP-Router# sh proc cpu | i CPU|PID|BGP
CPU utilization for five seconds: 93%/2%; one minute: 32%; five minutes: 22%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
319 2319628 11589466 200 0.15% 0.05% 0.04% 0 BGP Router
320 568684 2305861 246 0.00% 0.01% 0.00% 0 BGP I/O
321 246815548 1497615 164807 76.47% 9.23% 6.50% 0 BGP Scanner
58

High Memory
L3-Switch# sh proc mem | i PID|BGP
PID TTY Allocated Freed Holding GetbufsRetbufs Process
319 0 541682808 353471992 177441136 0 0 BGP Router
320 0 1377432 2361312 7048 0 0 BGP I/O
321 0 136 323920 10216 0 0 BGP Scanner
L3-Switch# shipbgp sum
Neighbor V AS MsgRcvdMsgSentTblVerInQOutQ Up/Down State/PfxRcd
32.124.75.251 4 209 1741759 68344 9564122 0 0 6w5d 251577
52.111.238.129 4 5555 2798645 68231 9564122 0 0 1w2d 254104
192.0.1.148 4 22222 68448 2134480 9564122 0 0 3w3d 35
192.0.2.228 4 33333 67386 2381477 9564122 0 0 5d01h 118
192.0.3.254 4 11111 2140027 2272911 9564130 0 0 6w5d 254360
750K routes (if no soft-reconfig)
1.5M routes (if soft-reconfig)
542MB of memory for BGP
59

LatencyPerception v. Reality
What could cause this horrible latency???
Reply from 209.85.171.100: bytes=32 time=5ms TTL=247
60
BGP scanner process takes higher priority than ICMP processing. Move on, nothing to see here.

Section ReviewTroubleshooting
What are 3 reasons that could cause peer establishment problems?
What are the advantages and disadvantages of soft reconfiguration?
What is required in ordered to announce a prefix?
What kind of information can you get from the looking glass route servers?
61

BGP Resources
North American Network Operators Group (NANOG)
http://www.nanog.org
www.traceroute.org
62

http://www.slideshare.net	40
http://jeeveshwarni.blogspot.com	3
http://senaintro.blackboard.com	2
http://translate.googleusercontent.com	1
http://static.slidesharecdn.com	1

Implementing Internet and MPLS BGP

by Private

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 47

Statistics

Implementing Internet and MPLS BGP Presentation Transcript