Presented at "FIBRE-BR Camp" in Ouro Preto (MG), Brazil. Apr.2012

1. Study for FIBRE-BR Backbone
Network Architecture
Alex S. Moura
FIBRE-BR Camp, 28-29 April 2012
Ouro Preto (MG), Brazil
1

2. RNP Backbone Physical Topology
• Agreement with local
telco Oi, brokered by
regulatory agency,
Anatel
• 3Gbps and 10 Gbps
capacities in 24 of 27
state capitals
• Currently no fibre to last
3 capitals. (Maybe in
2013?)
•North: 2 terrestrial links
and 1 satellite link
2
2

3. RNP Backbone Logical Topology
RNP Panorama: http://www.rnp.br/en/traffic/weathermap.php
3
3

4. RNP Backbone Core Equipments
Backbone Core Equipments PoPs: Distribution and access equipments
Juniper MX-480 (25)
4 to 15x 10GigE
20x 1GigE
Extreme BlackDiamond 8810
Brocade Netiron MLX-4
Juniper M10i
M10i
Nx 1GbE
Nx E1 Cisco 7206VXR
Juniper M7i Extreme Networks X-450a
Nx 1GbE
Nx E1
Nx STM-1
Cisco 6509
4
4

5. FIBRE Testbed Facility
FIBRE-BR Testbed Service Premises
• Service should be simple to deploy and operate in the environment of PoP
operators, and easy to use by project researchers and end users
• Preferably the complexity should be left on the FIBRE-BR side of the island of
the demarcation point
• Implementation in RNP backbone PoPs should not require tools or
processes that demand significant amount of investment in human
resources for its operation
5
5

6. FIBRE Testbed Facility
FIBRE-BR Testbed Service Premises (cont.)
• Service will must have maximum bandwidth enforced in RNP Backbone,
PoPs networks and in client organizations infrastructures in order to not
cause negative impact in production traffic
• Parts of the FIBRE network can be shutdown by RNP NOC operators if
needed
• The service may not have infrastructure redundancy (high availability)
• Users must comply with service premises and restrictions
6
6

7. RNP Backbone: Infrastructure for FIBRE-BR
Lack of diversity of fibers, lambdas and equipments lead to design of
an logical overlay infrastructure
• Allows deployment of DCN service and testbeds
“Virtual backbone” using same equipments of physical backbone
that runs the IP network by use of virtualization technology
• Approach used for offer dedicated infrastructure for RNP’s DCN service
• Logical topologies over physical topology
Deployment of virtual backbone made by RNP’s engineering and
operations team
7
7

8. RNP Backbone: Infrastructure for FIBRE-BR
Physical connections and rings
Fiber “virtual” path AP
MA PA
AC CE RN
RO PB-JP
TO AM
PB-CG
MT GO
PE PI
MS
DF MG • No
mul(ple
AL
lambdas
RJ ES BA SE
available
PR SP RR
• Transparency
to
codifica(on
and
RS SC bandwidth
8
8

9. RNP Backbone: Infrastructure for FIBRE-BR
Physical connections and rings
Fiber “virtual” path AP
MA PA
AC CE RN
RO PB-JP
TO AM
PB-CG
MT GO
PE PI
MS
DF MG • No
mul(ple
AL
lambdas
λ RJ ES BA SE
Routers
available
Layer 3 (IP)
Logical Systems • Transparency
to
PR SP RR (virtual routers)
Layer 2 (circuits) codifica(on
and
RS SC bandwidth
9
9

10. FIBRE-BR Topology: Proposal #1
10
10

11. RNP Backbone: Infrastructure for FIBRE-BR
PoP A PoP B PoP C
Router Router Router
10GE 10GE
Backbone VLAN Backbone VLAN Backbone VLAN
PoP Access GbE PoP Access GbE PoP Access GbE
PoP Distribution PoP Distribution PoP Distribution
Router / Switch Router / Switch Router / Switch
Access Access Access Access 1GE 1GE
Client Client Client Client Client Client
11
11

12. Proposal for FIBRE-BR Architecture
A 10,000 Feet View
12
12

13. FIBRE-BR Experiments requeriments
The FIBRE-BR should allow
• Perform networking experiments in parallel (how many?)
− (There cannot be any interference or conflicts between
experiments)
• Each experiment should be allowed to choose resources: servers,
interfaces etc.
• Each experiment should be allowed to define the network topology (?)*
• Acess to control of selected resources
• Allow acess to measurements and statistics from the experiment
• Some facility to reproduce each experiment
13
13

14. FIBRE-BR Backbone requeriments and some questions
• Should allow communication in Layer 2 between 10 islands of
resources in geographically dispersed end sites in Brazil
• Some questions:
• How important is the High Availability (HA) for this infrastructure?
• Reminder: backbone links failures do happen.
14
14

15. FIBRE-BR Backbone requeriments and some questions (cont.)
• Some questions (cont.):
• Does the FIBRE-BR should allow to each experiment to “build” its own
topology “mapped over” (strict) the physical RNP backbone paths?
• In other words, is there a need to allow each experiment to define it’s own
logical network topology “in sync” with the physical backbone topology?
15
15

16. Questions About Experiments Topologies
• Actual CMFs do not allow automatic control to create arbitrary network
topologies
• Physical topology (physical backbone) x Logical topology (logical FIBRE
backbone)
• Knowledge of the geographic locations of each island wouldn’t be
enough for each experiment (at least in the initial project phase)?
• Does the presence of OpenFlow switches in RNP PoPs of the "physical
backbone" which does not have connected islands can change any
requerement of the physical or logical topology of FIBRE experiments?
16
16

17. Questions About Experiments Topologies (cont.)
Some facts:
• The offering of automatic definition and configuration of a arbitrary
topology for each experiment will demand efforts in development - make
changes - in some actual CMF
• Up to where was possible to verify, this capacity is not offered in any
testbeds environment in production nowadays (this affirmation can
require a confirmation from experts)
17
17

18. Questions About Experiments Topologies (cont.)
Complexity x Pragmatism
• Is it really necessary for the FIBRE to offer automatic arbitrary topology
configuration geographically in sync with the physical backbone topology
for the experiments since day one?
− What kind of experiments would require this capability?
− To obtain what kind of results?
18
18

19. Proposal for FIBRE-BR Network Architecture
Proposal: Deploy the FIBRE Backbone in phases
• Phase 1 - Deploy a "full mesh" topology using VPLS interconecting all PoPs
that connect FIBRE-BR islands
• Phase 2 - Extend Phase 1 topology to an “hybrid topology”, with some
circuits with explicitly defined paths over the physical RNP backbone,
allowing some “determinism” in part of the FIBRE-BR topology
• Phase ...
• Phase “N” - Deploy a CMF with capability to control all FIBRE and
experiments topologies automatically.
19
19

20. Proposal for the FIBRE-BR Backbone
Proposal for Phase 1
Deploy a "full mesh" topology using VPLS to
interconect all FIBRE-BR islands
20
20

21. FIBRE Testbed Facility (cont.)
Option 1
• 1 or 2 dedicated physical 1GigE downlink interfaces from Juniper’s routers to dedicated box
(router or switch) in the PoPs that connect FIBRE islands
• Interconnection of all interfaces in 1 or 2 L2 networks using VPLS (MPLS Virtual Private LAN
Service)
• The VPLS can be configured in two fashions:
− “Loose”: the interconnections of end nodes using MPLS not necessarily will go through
specific paths over the physical layer backbone and can be rerouted dinamically in case
of a physical layer topology failure (H.A.)
− “Strict”: the interconnections of end nodes using MPLS will go through explicitly defined
paths over the physical layer backbone and can be rerouted dinamically in case of a
physical layer topology failure (H.A.)
Option 2
• Use of Juniper JUNOS Logical Systems feature
• 1 or 2 dedicated VPLS Layer 2 networks between logical systems’ interfaces
21
21

22. RNP Backbone: Infrastructure for FIBRE-BR
PoP A PoP B PoP C
Router Router Router
10GE 10GE
Backbone VLAN Backbone VLAN Backbone VLAN
FIBRE MPLS FIBRE MPLS
CLOUD CLOUD
Dedicated
physical
FIBRE
BACKBONE
Experiment VPLS 1GE
router Experiment VPLS 1GE Experiment VPLS 1GE
interface
Regular router Regular router Regular router
or switch or switch or switch
1GE 1GE 1GE 1GE 1GE 1GE
Common Island Use Case Common Island Use Case Common Island Use Case
ISLAND
22
22

23. RNP Backbone: Infrastructure for FIBRE-BR
PoP A PoP B PoP C
Router Router Router
10GE 10GE
Backbone VLAN Backbone VLAN Backbone VLAN
Logical System FIBRE VMAN Logical System FIBRE VMAN Logical System
Dedicated
physical
FIBRE
BACKBONE
Experiment VLANs 1GE
router Experiment VLANs 1GE Experiment VLANs 1GE
interface
OpenFlow Box OpenFlow Box OpenFlow Box
Legacy Flow Legacy Flow Legacy Flow
RouteFlow RouteFlow RouteFlow
1GE 1GE 1GE 1GE 1GE 1GE
Common Island Use Case Common Island Use Case Common Island Use Case
ISLAND
23
23

24. RNP Backbone: Infrastructure for FIBRE-BR
PoP A PoP B PoP C
Router Router Router
10GE 10GE
Backbone VLAN Backbone VLAN Backbone VLAN
Logical System FIBRE VMAN Logical System FIBRE VMAN Logical System
Dedicated
physical
Experiment VLANs 1GE
router Experiment VLANs 1GE Experiment VLANs 1GE
interface
FIBRE
BACKBONE
Common Common Common
Router or Switch Router or Switch Router or Switch
OpenFlow Box OpenFlow Box OpenFlow Box
Legacy Flow Legacy Flow Legacy Flow
RouteFlow RouteFlow RouteFlow
1GE 1GE 1GE 1GE 1GE 1GE
Common Island Use Case Common Island Use Case Common Island Use Case
ISLAND 24
24

25. Thank you
Obrigado
Alex S. Moura
alex@rnp.br
25