Architecture – Lego
Introduction
TPM A&O
Responsible for the Orchestration
platform and application landscape of
the network department
peter.boers@surf.nl
Peter Boers
TPM NFV
Responsible for the NFV platform and
SURFfirewall service. Working on VPP
and faster packet processing on
normal hardware
eyle.brinkhuis@surf.nl
Eyle Brinkhuis
Productmanager A&O
Previously architect of SURFnet8 and
responsible for the Network
dashboard and Orchestration
platform
wouter.huisman@surf.nl
Wouter Huisman
Network Architecture building blocks
The fundamentals
The SURF network at it’s
core
Composing blocks
Orchestrating blocks and
visualising in the Network
dashboard
Composed services
Using all lego bricks to build
composed services across
multiple domains
Topology
The topology of the network describes the fiber layout and locations of all PoPs.
Hardware
The chassis and interfaces that build the connectivity between all PoPs
Protocols
The way a network is logically configured. This manages failover mechanisms and how traffic
flows across the fabric
Software
The systems that interact with the network or store relevant configuration data about the
network
Processes
The way humans interact with the network
What building blocks does a network have?
• Around 13000 km dark fiber
• Relatively static optical topology
• Still adding PoPs
• Around +/- 300 PoPs throughout Europe
• Optimised for transport to Amsterdam
• Plenty of capacity to scale by using more λ
• 10G backbone to 100G backbone
• Internationally upgrading to 400G
Topology
From Ciena to Juniper and more
• Standardisation on the Juniper MX portfolio of routers
• MX2008/MX10008 Amsterdam
• MX960 Core
• MX480 Core/Metro
• MX240 Metro/Access
• MX204 Access
• MX304 International high capacity
• 400G access capability
• Lenovo SR635 – NFV
• Fortigate 601e - Firewall
Hardware
MX2008
MX10008
MX960 MX480 MX240
MX204
MX304
Lenovo SR635
Protocols
SR-MPLS
From relatively static PBB-TE to dynamic SR-MPLS fully dynamic control plane
IS-IS
IGP to weight links and distribute labels
TI-LFA
Automatic failover calculation programmed into the PFE
EVPN
More capabilities compared to virtual switches, e.g. ESI’s
VRF
VRF’s can be dropped at the access
NMS
Due to the dynamic nature of the new network a different style
of NMS was needed. The NMS no longer needed any
knowledge about the topology. Just the endpoints.
Orchestrator
Provisioning of the network would only be done by Software,
we no longer use the cli to provision network elements
Integration
Operational and business support systems are tightly
integrated with the network
Innovation
Software is increasingly the driver of innovation
Software
It’s no longer about making config work, but creating and end-to-end service
portfolio.
• Self-Service
• Network is a facilitator of end-to-end services
• All changes are standardised
• Less manual work
• An increase in dependence on software
• Reliable and repeatable changes
• Portfolio will be simplified to reflect the lego blocks
Processes
The fundamentals
• Each service that we create uses underlying resources described in one of
these categories
• These resources are the “lego bricks” that encompass the SURF network
• The lego bricks working together result in a wider variety of services and a
more diverse portfolio
• The network and NFV platform can also be seen as a lego bricks within the
SURF organisation
• The A&O platform is the network departments interface to the wider world
and the teeth to which other ”lego bricks” can attach.
Composing blocks
Where we came from
3 tiered network
1. Optical – managed with a
controller
2. Carrier Ethernet – managed with a
controller/NMS
3. IP-core – completely by hand
All supporting systems by hand (IPAM,
DNS, documentation/CMDB
Engineers had to provision a service
into A LOT of different systems
A network change
• Manual work for upto half a day
• Generating ID’s for all services
• Reserving IP’s in IPAM
• Registering everything in DNS
• Documenting in IMS
• Configuring the network
• Resulted in
• Mistakes
• Configuration inconsistencies
• A network of configuration, not of services
Why automation & …
Eliminate repetitive
& time consuming
tasks
Prevent human
mistakes
Automation != Orchestration
Automation
START
END
PLAN
DO
CHECK
ACT
Orchestration
Why automation & orchestration…
Eliminate repetitive
& time consuming
tasks
Prevent human
mistakes
Up-to-date
service lifecycle
Enable self service AI
Customer dashboard
Architecture in use at SURF
Open sourced https://workfloworchestrator.org
Producten catalogus
Lifecycle of a service
Subscription
of product X
“a service is an instance
of a product, and called
subscription”
Create WF
product X
Modify WF
product X
Validate WF
product X
Executed daily
Terminate WF
product X
Workflow Engine
WORKFLOW
Process
Input
form(s)
Step 1 Step 2 Step 3 Step 4 Step 5 Step 6
Each Step writes the state to the database and is used as input for the
next step
Each (atomic) Step can be retried, making the workflow robust
GUI orchestrator - workflows
GUI orchestrator - processes
Network dashboard
• Build on single source of truth
of orchestrator
• Influx data for traffic graph and
SLS
• FW stats
• Planned work notifications
• But also
• SURFcert
• Vulnerabilities & DDoS
• SURFwireless
• SURFdomeinen (end 2023)
Beschi
kbaar
voor
“veran
twoor
delijke
” rol
Doorlo
op
Step
up
authe
nticati
e
Beschi
kbare
acties
Self service in Netwerk dashboard
• Beschikbaar voor “verantwoordelijke” rol
• Doorloop Step up authenticatie
• Beschikbare acties
• DDoS filter
• Automitigation filter
• Speed policer
• VLAN aanpassen
• BGP priority
• IP prefix toevoegen
• L2VPN uitbreiden
demo
Firewall product – composed product
SURFinternet
L2VPN
L2VPN
SURFinternet
SURFinternet
L2VPN
L2VPN L2VPN
L2VPN
L2VPN
L3VPN
Firewall opgebouwd uit de producten catalogie
Firewall met 1 klant poort gekoppeld aan SURFinternet
Firewall opgebouwd uit de producten catalogie
Firewall met 4 klant poorten gekoppeld aan SURFinternet
Firewall opgebouwd uit de producten catalogie
Firewall met 4 klant poorten gekoppeld aan SURFinternet
en gekopppeld met een L3VPN, bijv naar Azure Express Route
Composed services
SURFfirewall
Built upon several building blocks:
- SURFinternet
- L2VPN
- L3VPN
- FW
Usable in any configuration
Physical firewalls in central location
Usecases:
- Routers
- VPN concentrators
- Wireless controllers
What about other services?
NFV technology domain
- Handles compute stuff
- In-house developed
- Based on ETSI
NFV-MANO & NFVi
Payload from orchestrator
- Service version
- Availability_zone
- State
- Identifier
Customer
information
Update project
ticket
Reserve p2p
prefixes
Create IP-gateways
Create circuits
Create circuits
for impact/monitoring
Create firewall
Request license
Configure connectivity
Prepare Fortimanager
Validate everything
Put in sync
Future work
- Integrate with SURF HPC services
- Cloud Providers
Roadmap
Deploy SURFnet 8
with A&O
Integrate Network
Dashboard
Composed services
SURF HPC resources
NFV-services
Cloud provider
integration
Demo at central square!
Want to see it happening?
Eyle Brinkhuis
Eyle.Brinkhuis@surf.nl
Linkedin.com/in/eyle

SURF Lego - Architecture - Peter Boers- NWD23

  • 1.
  • 2.
    Introduction TPM A&O Responsible forthe Orchestration platform and application landscape of the network department peter.boers@surf.nl Peter Boers TPM NFV Responsible for the NFV platform and SURFfirewall service. Working on VPP and faster packet processing on normal hardware eyle.brinkhuis@surf.nl Eyle Brinkhuis Productmanager A&O Previously architect of SURFnet8 and responsible for the Network dashboard and Orchestration platform wouter.huisman@surf.nl Wouter Huisman
  • 3.
    Network Architecture buildingblocks The fundamentals The SURF network at it’s core Composing blocks Orchestrating blocks and visualising in the Network dashboard Composed services Using all lego bricks to build composed services across multiple domains
  • 4.
    Topology The topology ofthe network describes the fiber layout and locations of all PoPs. Hardware The chassis and interfaces that build the connectivity between all PoPs Protocols The way a network is logically configured. This manages failover mechanisms and how traffic flows across the fabric Software The systems that interact with the network or store relevant configuration data about the network Processes The way humans interact with the network What building blocks does a network have?
  • 5.
    • Around 13000km dark fiber • Relatively static optical topology • Still adding PoPs • Around +/- 300 PoPs throughout Europe • Optimised for transport to Amsterdam • Plenty of capacity to scale by using more λ • 10G backbone to 100G backbone • Internationally upgrading to 400G Topology
  • 6.
    From Ciena toJuniper and more • Standardisation on the Juniper MX portfolio of routers • MX2008/MX10008 Amsterdam • MX960 Core • MX480 Core/Metro • MX240 Metro/Access • MX204 Access • MX304 International high capacity • 400G access capability • Lenovo SR635 – NFV • Fortigate 601e - Firewall Hardware MX2008 MX10008 MX960 MX480 MX240 MX204 MX304 Lenovo SR635
  • 7.
    Protocols SR-MPLS From relatively staticPBB-TE to dynamic SR-MPLS fully dynamic control plane IS-IS IGP to weight links and distribute labels TI-LFA Automatic failover calculation programmed into the PFE EVPN More capabilities compared to virtual switches, e.g. ESI’s VRF VRF’s can be dropped at the access
  • 8.
    NMS Due to thedynamic nature of the new network a different style of NMS was needed. The NMS no longer needed any knowledge about the topology. Just the endpoints. Orchestrator Provisioning of the network would only be done by Software, we no longer use the cli to provision network elements Integration Operational and business support systems are tightly integrated with the network Innovation Software is increasingly the driver of innovation Software
  • 9.
    It’s no longerabout making config work, but creating and end-to-end service portfolio. • Self-Service • Network is a facilitator of end-to-end services • All changes are standardised • Less manual work • An increase in dependence on software • Reliable and repeatable changes • Portfolio will be simplified to reflect the lego blocks Processes
  • 10.
    The fundamentals • Eachservice that we create uses underlying resources described in one of these categories • These resources are the “lego bricks” that encompass the SURF network • The lego bricks working together result in a wider variety of services and a more diverse portfolio • The network and NFV platform can also be seen as a lego bricks within the SURF organisation • The A&O platform is the network departments interface to the wider world and the teeth to which other ”lego bricks” can attach.
  • 11.
  • 12.
    Where we camefrom 3 tiered network 1. Optical – managed with a controller 2. Carrier Ethernet – managed with a controller/NMS 3. IP-core – completely by hand All supporting systems by hand (IPAM, DNS, documentation/CMDB Engineers had to provision a service into A LOT of different systems
  • 13.
    A network change •Manual work for upto half a day • Generating ID’s for all services • Reserving IP’s in IPAM • Registering everything in DNS • Documenting in IMS • Configuring the network • Resulted in • Mistakes • Configuration inconsistencies • A network of configuration, not of services
  • 14.
    Why automation &… Eliminate repetitive & time consuming tasks Prevent human mistakes
  • 15.
  • 16.
    Why automation &orchestration… Eliminate repetitive & time consuming tasks Prevent human mistakes Up-to-date service lifecycle Enable self service AI Customer dashboard
  • 17.
    Architecture in useat SURF Open sourced https://workfloworchestrator.org
  • 18.
  • 19.
    Lifecycle of aservice Subscription of product X “a service is an instance of a product, and called subscription” Create WF product X Modify WF product X Validate WF product X Executed daily Terminate WF product X
  • 20.
    Workflow Engine WORKFLOW Process Input form(s) Step 1Step 2 Step 3 Step 4 Step 5 Step 6 Each Step writes the state to the database and is used as input for the next step Each (atomic) Step can be retried, making the workflow robust
  • 21.
  • 22.
  • 23.
    Network dashboard • Buildon single source of truth of orchestrator • Influx data for traffic graph and SLS • FW stats • Planned work notifications • But also • SURFcert • Vulnerabilities & DDoS • SURFwireless • SURFdomeinen (end 2023)
  • 24.
    Beschi kbaar voor “veran twoor delijke ” rol Doorlo op Step up authe nticati e Beschi kbare acties Self servicein Netwerk dashboard • Beschikbaar voor “verantwoordelijke” rol • Doorloop Step up authenticatie • Beschikbare acties • DDoS filter • Automitigation filter • Speed policer • VLAN aanpassen • BGP priority • IP prefix toevoegen • L2VPN uitbreiden demo
  • 25.
    Firewall product –composed product SURFinternet L2VPN L2VPN SURFinternet SURFinternet L2VPN L2VPN L2VPN L2VPN L2VPN L3VPN
  • 26.
    Firewall opgebouwd uitde producten catalogie Firewall met 1 klant poort gekoppeld aan SURFinternet
  • 27.
    Firewall opgebouwd uitde producten catalogie Firewall met 4 klant poorten gekoppeld aan SURFinternet
  • 28.
    Firewall opgebouwd uitde producten catalogie Firewall met 4 klant poorten gekoppeld aan SURFinternet en gekopppeld met een L3VPN, bijv naar Azure Express Route
  • 29.
  • 30.
    SURFfirewall Built upon severalbuilding blocks: - SURFinternet - L2VPN - L3VPN - FW Usable in any configuration Physical firewalls in central location
  • 31.
    Usecases: - Routers - VPNconcentrators - Wireless controllers What about other services?
  • 32.
    NFV technology domain -Handles compute stuff - In-house developed - Based on ETSI NFV-MANO & NFVi
  • 34.
    Payload from orchestrator -Service version - Availability_zone - State - Identifier
  • 35.
    Customer information Update project ticket Reserve p2p prefixes CreateIP-gateways Create circuits Create circuits for impact/monitoring Create firewall Request license Configure connectivity Prepare Fortimanager Validate everything Put in sync
  • 36.
    Future work - Integratewith SURF HPC services - Cloud Providers
  • 37.
    Roadmap Deploy SURFnet 8 withA&O Integrate Network Dashboard Composed services SURF HPC resources NFV-services Cloud provider integration
  • 38.
    Demo at centralsquare! Want to see it happening?
  • 39.