SlideShare a Scribd company logo
1 of 63
Download to read offline
How our Cloudy Mindsets
Approached Physical Routers
SNMP was not an option
Steffen Gebert
DENOG12, 09.11.2020
After the latest project, EMnify became a 99% only cloud
company. To meet growing scalability and reliability
requirements of the interconnection between our AWS-
based deployments and multiple carriers, BGP peerings
had to be moved out of AWS. Therefore, a pair of Juniper
routers were put into place. For a company fully relying on
cloud services so far, this alien technology resulted in
several challenges.
We want to share, how we solved the integration puzzle of
this physical equipment into our existing workflows and
tools. The use of CI/CD systems for applying changes,
AWS CloudWatch, Prometheus and Grafana for
monitoring as well as the reluctance to run applications
that require a lot of shepherding lead our research to find
the right glue - the glue between these pieces of iron and
our cloud infrastructure.
Being used to CI/CD processes backed by automated tests,
we wanted to adapt these practices here as well. As a
result, configuration changes are rolled out by an
automated pipeline using Ansible. Efforts for automated
testing were made, where we failed. We explain why and
what we did instead as well as what we envision for the
future.
As every other part of our system, we want its monitoring
data accessible via Grafana.
With the help of pmacct and fluentbit, we can treat IPFIX
flow records as they were logs. With the help of jtimon,
Prometheus stores the routers’ metrics as we are used to
do, in doubt tickled out through few custom YANG models.
In summary, the integration worked very well, while we still
have several learnings and pain points to share.
Abstract
Thanks to our Sponsors!
Cloudy Mindset?
5 years ago
All rights reserved | Confidential
10.
11.
3-10 years ago
6@StGebert
Since 2017
7@StGebert
Is This a Better World?
Focus on Business Value
Prefer Managed Services
And Suddenly… Hardware?
Agenda
Context
Deploy-
ment
Moni-
toring
EMnify’s IoT Connectivity Platform
13@StGebert
Cellular connectivity
in 500+ networks in
185 countries
RESTful APIs Pay as you go
pricing
SMS/USSD to REST
bridge
Secure connectivity
via VPN and AWS
natively
Implemented using
own virtualized
mobile core network
Supporting Global IoT Deployments
Home-routing of roaming SIM data
prevents distributed architecture
EMnify’s mobile core network is
deployed in multiple AWS regions
– keeping data local
@StGebert 14
GRX/IPX Network (GPRS Roaming Exchange)
15
Internet
Telco world
GRX/IPX
@StGebert
Our scale[throughput] bores you
We’re critical to our customers’ success
Increased demands vs. AWS as
“General Purpose Cloud”
Running BGP on AWS?
We had to move logic out of AWS
We could not find a fitting
managed service
We had to get hardware
We chose boring technology
Greenfield project
• Juniper MX204
• Colocation rack space
• Fiber links towards
2 carriers
AWS
• Out-of-band management
access via OPNsense
25@StGebert
Setup – Twice per region
Integration Points
26
Metrics
Syslog
Flows records
Alerts
Config
@StGebert
27
Design Principles
80/20 rule
aka
MVP
Don’t get out
of our comfort
zone
Don’t setup
anythat that
requires lot of
handholding
@StGebert
Deployment
A human shall not SSH into
something
M Y I N N E R S E L F
juniper_junos Ansible Modules
@StGebert 30
Configuration Deployment
Render
Config
•Single config file
•Parametrized using variables
Commit
•Upload using juniper_junos_config
•Load override mode
•Issue (to be) confirmed commit
Test
•Ping tests
Confirm
•Confirm commit
@StGebert 31
Ansible Playbook - Code Example
@StGebert 32
- name: install generated configuration file onto device
juniper_junos_config:
provider: "{{ juniper_connection_settings }}"
src: "{{ conf_file }}"
load: override
comment: "playbook execution, commit confirmed"
confirmed: 3 # wait X minutes until rollback
diff: yes
ignore_warning: yes
register: config_results
notify: confirm previous commit
• Separate AWS account
• Isolated connectivity
33@StGebert
Config Pipeline
AWS CodePipeline
AWS CodeBuild
And where are the tests?
In a Perfect World..
35
• 2-star review could have been mine :D
• Latest version: 18.4R1
• Takes ~30min to be ready
• AWS does not support VLANs!
• Only for manual testing
• Maybe eve-ng or GNS3 could help?
@StGebert
• Start virtualized topology in network emulator
• Apply configuration pipeline
• Emulate BGP peers
• Execute end-to-end connectivity tests
• Emulate link failures
• Verify connectivity
• AWS: run on bare metal host (b/c CPU VMX)
On My Bucket List
36@StGebert
Routine Operations (Runbooks)
38@StGebert
Firmware Update - Checks
/workspace/prod # ansible-playbook upgrade_check.yaml -u steffen.gebert
...
TASK [Validate result] *****************
[ mx204-am3 ] Chassis Alarms
----------------------------
Expect:
No alarms currently active
Actual:
No alarms currently active
...
[ mx204-am3 ] Core Dumps
----------------------------
Expect:
/var/crash/*core*: No such file or directory
Actual:
/var/crash/*core*: No such file or directory
[ mx204-am3 ] ⚠ Proceed? ⚠
:
Press 'C' to continue the play or 'A' to abort
39@StGebert
Firmware Update - Draining
- name: Drain traffic
juniper_junos_config:
provider: "{{ juniper_connection_settings }}"
load: 'set'
lines:
- 'activate policy-options policy-statement OUT-OF-SERVICE-SWITCH term as-path-pr
comment: 'Drain traffic to router for upgrade'
- name: Traffic drained
pause:
prompt: |
[ {{item}} ] Traffic is draining.
Verify that traffic is completely drained on the following dashboard before proce
[ {{item}} ] ⚠ Proceed with the JunOS upgrade ⚠?
loop: "{{ ansible_play_hosts }}"
40@StGebert
Firmware Update – Execute!
- name: Install Junos OS package
juniper_junos_software:
provider:
host: "{{ ansible_host }}"
timeout: 3600
remote_package: "{{ junos_vm_file }}"
validate: True
cleanfs: False
vmhost: True
reboot: True
ignore_errors: yes # rpc times out when upgrading, despite the provider timeout setti
register: output
Deploy a file
¯_(ツ)_/¯
Challenges
Max length
of file
copy URLs
Feedback for
invalid confg
Amount of
boilerplate
code
Monitoring
Syslogs
•Who logged into the router?
•What’s happening in the router?
44@StGebert
Syslog Implementation
rsyslog
Amazon CloudWatch
Logs
1 containers
to run
Flow Records
Flow Records
Network-to-
Network
Interface
(GTP traffic)
~20k parallel
flows
1 flow =>
1 log line?!
46@StGebert
Amazon CloudWatch
Logs
•How much traffic per AS?
•Did we receive any signaling from XYZ and did we really respond?
47@StGebert
Flow Records Collection
IPFIX
2 containers
to run
BMP
FireLens
pmacct / nfacct
@StGebert 48
libpcap
NFLOG
NetFlow
IPFIX BGP
BMP
RPKI
tag
filter
aggregate
split
RDBMS
NoSQL
MQ
NetFlow
Memory
File
stdout
protocol
Enrichment
@StGebert 49
host_ip src_ip dst_ip src_port dst_port src_asn dst_asn src_ifindex dst_ifindex …
direction
organization
hostname carrier
drop
BGP & BFD
Lua Magic
@StGebert 50
-- Sets GTP-c or GTP-u protocol depending on port numbers
function setGTPProtocol(tag, timestamp, record)
local code = 0
local gtp_ports = {
["GTP-c"] = 2123,
["GTP-u"] = 2152,
["GTP'"] = 3386,
}
local new_record = record
for protocol, port in pairs(gtp_ports)
do
if record["source.port"] == port or record["destination.port"] == port then
new_record["network.application"] = "GTP"
new_record["network.protocol"] = protocol
code = 2
end
end
return code, timestamp, new_record
51@StGebert
52@StGebert
Inbound traffic by AS query
@StGebert 53
fields concat(source.as.organization.name, ', ’,
source.as.organization.country, ' (AS ', source.as.number, ')') as org
| filter @logStream = "flows"
| filter host.name like /^$host$/
| filter concat(source.as.number, ' ', source.as.organization.name, ' ',
source.as.organization.country,' ',source.as.organization.tadig) like /$operator/
OR concat(destination.as.number, ' ', destination.as.organization.name, ' ’,
destination.as.organization.country, ' ', destination.as.organization.tadig)
like /$operator/
| filter network.peer.destination.as.organization.name like /^$carrier$/
| filter network.direction = "inbound"
| filter network.protocol like /$protocol/
| filter 10000
| stats sum(network.bytes)/60*8 as `` by org,bin($time_interval)
| sort `` desc
CloudWatch
Read Limits
Challenges
CloudWatch
Write Limits
pmacct
config
“creativity”
Metrics
Metrics Demand
56
Temperature, light
levels, etc.
State, throughput,
errors, etc.
State, prefixes
received/accepted/installed
Hard-
ware
Inter-
faces
BGP
@StGebert
Prometheus
• High cardinality, high frequency metrics collection
57@StGebert
Metrics Implementation
Junos Telemetry
(JTI)
1 container
to run
JTIMON
58@StGebert
Metrics Examples
JTI Sensor
availability
Challenges
JTIMON
config file
duplication
JTIMON ENUM
support
PKI setup
Alerting
Prometheus
• Prometheus-integrated alerting
61@StGebert
Alerting Implementation
Junos Telemetry
(JTI)
JTIMON
Alertmanager
• Integrated hardware into an otherwise fully cloud-based environment
• Avoid new processes
• Avoid new (user-facing) tooling
• Found tooling to bridge gaps to “what we’re comfortable with”
• 1-2 containers running existing open source tooling
• No guarantuee that this scales to 10s of devices
• Please contact me, if you want details (configs etc.) or have suggestions!
62@StGebert
Summary & Conclusion
Need a
Lockdown
Project?
Go to emnify.com/devs

More Related Content

What's hot

Istio service mesh: past, present, future (TLV meetup)
Istio service mesh: past, present, future (TLV meetup)Istio service mesh: past, present, future (TLV meetup)
Istio service mesh: past, present, future (TLV meetup)elevran
 
Edge to Instance - AWS Networking
Edge to Instance - AWS Networking Edge to Instance - AWS Networking
Edge to Instance - AWS Networking Amazon Web Services
 
Istio - A Service Mesh for Microservices as Scale
Istio - A Service Mesh for Microservices as ScaleIstio - A Service Mesh for Microservices as Scale
Istio - A Service Mesh for Microservices as ScaleRam Vennam
 
The service mesh: resilient communication for microservice applications
The service mesh: resilient communication for microservice applicationsThe service mesh: resilient communication for microservice applications
The service mesh: resilient communication for microservice applicationsOutlyer
 
Confluent Private Cloud | Rohit Bakhshi, Staff Product Manager
Confluent Private Cloud | Rohit Bakhshi, Staff Product ManagerConfluent Private Cloud | Rohit Bakhshi, Staff Product Manager
Confluent Private Cloud | Rohit Bakhshi, Staff Product ManagerHostedbyConfluent
 
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...Daniel Oh
 
CDN_Netflix_analysis
CDN_Netflix_analysisCDN_Netflix_analysis
CDN_Netflix_analysisSanket Jain
 
Open-IX Presentation: Datacenter Selection by Adam Rothschild
Open-IX Presentation: Datacenter Selection by Adam RothschildOpen-IX Presentation: Datacenter Selection by Adam Rothschild
Open-IX Presentation: Datacenter Selection by Adam RothschildPacket
 
Megaport Enabled AWS Direct Connect
Megaport Enabled AWS Direct ConnectMegaport Enabled AWS Direct Connect
Megaport Enabled AWS Direct ConnectDavid McCullough
 
Why you should have a Schema Registry | David Hettler, Celonis SE
Why you should have a Schema Registry | David Hettler, Celonis SEWhy you should have a Schema Registry | David Hettler, Celonis SE
Why you should have a Schema Registry | David Hettler, Celonis SEHostedbyConfluent
 
Kafka Summit 2019 Microservice Orchestration
Kafka Summit 2019 Microservice OrchestrationKafka Summit 2019 Microservice Orchestration
Kafka Summit 2019 Microservice Orchestrationlarsfrancke
 
L’odyssée d’une requête HTTP chez Scaleway
L’odyssée d’une requête HTTP chez ScalewayL’odyssée d’une requête HTTP chez Scaleway
L’odyssée d’une requête HTTP chez ScalewayScaleway
 
Sdn users group_january_2016v5
Sdn users group_january_2016v5Sdn users group_january_2016v5
Sdn users group_january_2016v5Joel W. King
 
AWS Connectivity, VPC Design and Security Pro Tips
AWS Connectivity, VPC Design and Security Pro TipsAWS Connectivity, VPC Design and Security Pro Tips
AWS Connectivity, VPC Design and Security Pro TipsShiva Narayanaswamy
 
Secure Infrastructure Provisioning with Terraform Cloud, Vault + GitLab CI
Secure Infrastructure Provisioning with Terraform Cloud, Vault + GitLab CISecure Infrastructure Provisioning with Terraform Cloud, Vault + GitLab CI
Secure Infrastructure Provisioning with Terraform Cloud, Vault + GitLab CIMitchell Pronschinske
 
(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC
(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC
(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPCAmazon Web Services
 
Kubernetes Apache Kafka
Kubernetes Apache KafkaKubernetes Apache Kafka
Kubernetes Apache Kafkaconfluent
 
Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration
Cloudify: Open vCPE Design Concepts and Multi-Cloud OrchestrationCloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration
Cloudify: Open vCPE Design Concepts and Multi-Cloud OrchestrationCloudify Community
 

What's hot (20)

Istio service mesh: past, present, future (TLV meetup)
Istio service mesh: past, present, future (TLV meetup)Istio service mesh: past, present, future (TLV meetup)
Istio service mesh: past, present, future (TLV meetup)
 
Edge to Instance - AWS Networking
Edge to Instance - AWS Networking Edge to Instance - AWS Networking
Edge to Instance - AWS Networking
 
Istio - A Service Mesh for Microservices as Scale
Istio - A Service Mesh for Microservices as ScaleIstio - A Service Mesh for Microservices as Scale
Istio - A Service Mesh for Microservices as Scale
 
The service mesh: resilient communication for microservice applications
The service mesh: resilient communication for microservice applicationsThe service mesh: resilient communication for microservice applications
The service mesh: resilient communication for microservice applications
 
Confluent Private Cloud | Rohit Bakhshi, Staff Product Manager
Confluent Private Cloud | Rohit Bakhshi, Staff Product ManagerConfluent Private Cloud | Rohit Bakhshi, Staff Product Manager
Confluent Private Cloud | Rohit Bakhshi, Staff Product Manager
 
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
 
CDN_Netflix_analysis
CDN_Netflix_analysisCDN_Netflix_analysis
CDN_Netflix_analysis
 
Open-IX Presentation: Datacenter Selection by Adam Rothschild
Open-IX Presentation: Datacenter Selection by Adam RothschildOpen-IX Presentation: Datacenter Selection by Adam Rothschild
Open-IX Presentation: Datacenter Selection by Adam Rothschild
 
Megaport Enabled AWS Direct Connect
Megaport Enabled AWS Direct ConnectMegaport Enabled AWS Direct Connect
Megaport Enabled AWS Direct Connect
 
Rethinking Cloud Proxies
Rethinking Cloud ProxiesRethinking Cloud Proxies
Rethinking Cloud Proxies
 
Why you should have a Schema Registry | David Hettler, Celonis SE
Why you should have a Schema Registry | David Hettler, Celonis SEWhy you should have a Schema Registry | David Hettler, Celonis SE
Why you should have a Schema Registry | David Hettler, Celonis SE
 
Kafka Summit 2019 Microservice Orchestration
Kafka Summit 2019 Microservice OrchestrationKafka Summit 2019 Microservice Orchestration
Kafka Summit 2019 Microservice Orchestration
 
L’odyssée d’une requête HTTP chez Scaleway
L’odyssée d’une requête HTTP chez ScalewayL’odyssée d’une requête HTTP chez Scaleway
L’odyssée d’une requête HTTP chez Scaleway
 
Sdn users group_january_2016v5
Sdn users group_january_2016v5Sdn users group_january_2016v5
Sdn users group_january_2016v5
 
AWS Connectivity, VPC Design and Security Pro Tips
AWS Connectivity, VPC Design and Security Pro TipsAWS Connectivity, VPC Design and Security Pro Tips
AWS Connectivity, VPC Design and Security Pro Tips
 
Secure Infrastructure Provisioning with Terraform Cloud, Vault + GitLab CI
Secure Infrastructure Provisioning with Terraform Cloud, Vault + GitLab CISecure Infrastructure Provisioning with Terraform Cloud, Vault + GitLab CI
Secure Infrastructure Provisioning with Terraform Cloud, Vault + GitLab CI
 
(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC
(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC
(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC
 
Kubernetes Apache Kafka
Kubernetes Apache KafkaKubernetes Apache Kafka
Kubernetes Apache Kafka
 
12 Factor App
12 Factor App12 Factor App
12 Factor App
 
Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration
Cloudify: Open vCPE Design Concepts and Multi-Cloud OrchestrationCloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration
Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration
 

Similar to How our Cloudy Mindsets Approached Physical Routers

NFD9 - Dinesh Dutt, Data Center Architectures
NFD9 - Dinesh Dutt, Data Center ArchitecturesNFD9 - Dinesh Dutt, Data Center Architectures
NFD9 - Dinesh Dutt, Data Center ArchitecturesCumulus Networks
 
Netsft2017 day in_life_of_nfv
Netsft2017 day in_life_of_nfvNetsft2017 day in_life_of_nfv
Netsft2017 day in_life_of_nfvIntel
 
P&G BT Global Services - LLD Final Revision Year 2008.
P&G BT Global Services - LLD Final Revision Year 2008.P&G BT Global Services - LLD Final Revision Year 2008.
P&G BT Global Services - LLD Final Revision Year 2008.Kapil Sabharwal
 
BitVisor Summit 8「3. AQC107 Driver and Changes coming to network API」
BitVisor Summit 8「3. AQC107 Driver and Changes coming to network API」BitVisor Summit 8「3. AQC107 Driver and Changes coming to network API」
BitVisor Summit 8「3. AQC107 Driver and Changes coming to network API」BitVisor
 
Ground-Cloud-Cloud-Ground - NAB 2022 IP Showcase
Ground-Cloud-Cloud-Ground - NAB 2022 IP ShowcaseGround-Cloud-Cloud-Ground - NAB 2022 IP Showcase
Ground-Cloud-Cloud-Ground - NAB 2022 IP ShowcaseKieran Kunhya
 
Scalable Enterprise Ready Neutron Networking with Nuage Networks
Scalable Enterprise Ready Neutron Networking with Nuage NetworksScalable Enterprise Ready Neutron Networking with Nuage Networks
Scalable Enterprise Ready Neutron Networking with Nuage NetworksScott Sneddon
 
KubeCon London 2016 Ronana Cloud Native SDN
KubeCon London 2016 Ronana Cloud Native SDNKubeCon London 2016 Ronana Cloud Native SDN
KubeCon London 2016 Ronana Cloud Native SDNRomana Project
 
Building CloudScale Networks - AWS Summit Sydney 2018
Building CloudScale Networks - AWS Summit Sydney 2018Building CloudScale Networks - AWS Summit Sydney 2018
Building CloudScale Networks - AWS Summit Sydney 2018Amazon Web Services
 
In-Memory Key Value Store (KVS) in FPGA for Ultra Low Latency and High Throug...
In-Memory Key Value Store (KVS) in FPGA for Ultra Low Latency and High Throug...In-Memory Key Value Store (KVS) in FPGA for Ultra Low Latency and High Throug...
In-Memory Key Value Store (KVS) in FPGA for Ultra Low Latency and High Throug...Tom Diederich
 
Building the SD-Branch using uCPE
Building the SD-Branch using uCPEBuilding the SD-Branch using uCPE
Building the SD-Branch using uCPEMichelle Holley
 
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)Ontico
 
Challenges of L2 NID Based Architecture for vCPE and NFV Deployment
Challenges of L2 NID Based Architecture for vCPE and NFV Deployment Challenges of L2 NID Based Architecture for vCPE and NFV Deployment
Challenges of L2 NID Based Architecture for vCPE and NFV Deployment Bangladesh Network Operators Group
 
Contrail Deep-dive - Cloud Network Services at Scale
Contrail Deep-dive - Cloud Network Services at ScaleContrail Deep-dive - Cloud Network Services at Scale
Contrail Deep-dive - Cloud Network Services at ScaleMarketingArrowECS_CZ
 
Introduction to Programmable Networks by Clarence Anslem, Intel
Introduction to Programmable Networks by Clarence Anslem, IntelIntroduction to Programmable Networks by Clarence Anslem, Intel
Introduction to Programmable Networks by Clarence Anslem, IntelMyNOG
 
2014/09/02 Cisco UCS HPC @ ANL
2014/09/02 Cisco UCS HPC @ ANL2014/09/02 Cisco UCS HPC @ ANL
2014/09/02 Cisco UCS HPC @ ANLdgoodell
 
Nfd18 anuta-networks
Nfd18 anuta-networksNfd18 anuta-networks
Nfd18 anuta-networksKiran Sirupa
 
Service Chaining - Cloud Network Services at Scale
Service Chaining - Cloud Network Services at ScaleService Chaining - Cloud Network Services at Scale
Service Chaining - Cloud Network Services at ScaleMarketingArrowECS_CZ
 

Similar to How our Cloudy Mindsets Approached Physical Routers (20)

NFD9 - Dinesh Dutt, Data Center Architectures
NFD9 - Dinesh Dutt, Data Center ArchitecturesNFD9 - Dinesh Dutt, Data Center Architectures
NFD9 - Dinesh Dutt, Data Center Architectures
 
Netsft2017 day in_life_of_nfv
Netsft2017 day in_life_of_nfvNetsft2017 day in_life_of_nfv
Netsft2017 day in_life_of_nfv
 
P&G BT Global Services - LLD Final Revision Year 2008.
P&G BT Global Services - LLD Final Revision Year 2008.P&G BT Global Services - LLD Final Revision Year 2008.
P&G BT Global Services - LLD Final Revision Year 2008.
 
BitVisor Summit 8「3. AQC107 Driver and Changes coming to network API」
BitVisor Summit 8「3. AQC107 Driver and Changes coming to network API」BitVisor Summit 8「3. AQC107 Driver and Changes coming to network API」
BitVisor Summit 8「3. AQC107 Driver and Changes coming to network API」
 
Where should I run my code? Serverless, Containers, Virtual Machines and more
Where should I run my code? Serverless, Containers, Virtual Machines and moreWhere should I run my code? Serverless, Containers, Virtual Machines and more
Where should I run my code? Serverless, Containers, Virtual Machines and more
 
Ground-Cloud-Cloud-Ground - NAB 2022 IP Showcase
Ground-Cloud-Cloud-Ground - NAB 2022 IP ShowcaseGround-Cloud-Cloud-Ground - NAB 2022 IP Showcase
Ground-Cloud-Cloud-Ground - NAB 2022 IP Showcase
 
Scalable Enterprise Ready Neutron Networking with Nuage Networks
Scalable Enterprise Ready Neutron Networking with Nuage NetworksScalable Enterprise Ready Neutron Networking with Nuage Networks
Scalable Enterprise Ready Neutron Networking with Nuage Networks
 
KubeCon London 2016 Ronana Cloud Native SDN
KubeCon London 2016 Ronana Cloud Native SDNKubeCon London 2016 Ronana Cloud Native SDN
KubeCon London 2016 Ronana Cloud Native SDN
 
Building CloudScale Networks - AWS Summit Sydney 2018
Building CloudScale Networks - AWS Summit Sydney 2018Building CloudScale Networks - AWS Summit Sydney 2018
Building CloudScale Networks - AWS Summit Sydney 2018
 
Open v ran
Open v ranOpen v ran
Open v ran
 
In-Memory Key Value Store (KVS) in FPGA for Ultra Low Latency and High Throug...
In-Memory Key Value Store (KVS) in FPGA for Ultra Low Latency and High Throug...In-Memory Key Value Store (KVS) in FPGA for Ultra Low Latency and High Throug...
In-Memory Key Value Store (KVS) in FPGA for Ultra Low Latency and High Throug...
 
Building the SD-Branch using uCPE
Building the SD-Branch using uCPEBuilding the SD-Branch using uCPE
Building the SD-Branch using uCPE
 
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)
 
Challenges of L2 NID Based Architecture for vCPE and NFV Deployment
Challenges of L2 NID Based Architecture for vCPE and NFV Deployment Challenges of L2 NID Based Architecture for vCPE and NFV Deployment
Challenges of L2 NID Based Architecture for vCPE and NFV Deployment
 
Contrail Deep-dive - Cloud Network Services at Scale
Contrail Deep-dive - Cloud Network Services at ScaleContrail Deep-dive - Cloud Network Services at Scale
Contrail Deep-dive - Cloud Network Services at Scale
 
Introduction to Programmable Networks by Clarence Anslem, Intel
Introduction to Programmable Networks by Clarence Anslem, IntelIntroduction to Programmable Networks by Clarence Anslem, Intel
Introduction to Programmable Networks by Clarence Anslem, Intel
 
2014/09/02 Cisco UCS HPC @ ANL
2014/09/02 Cisco UCS HPC @ ANL2014/09/02 Cisco UCS HPC @ ANL
2014/09/02 Cisco UCS HPC @ ANL
 
Nfd18 anuta-networks
Nfd18 anuta-networksNfd18 anuta-networks
Nfd18 anuta-networks
 
Gb over ip
Gb over ipGb over ip
Gb over ip
 
Service Chaining - Cloud Network Services at Scale
Service Chaining - Cloud Network Services at ScaleService Chaining - Cloud Network Services at Scale
Service Chaining - Cloud Network Services at Scale
 

More from Steffen Gebert

Building an IoT SuperNetwork on top of the AWS Global Infrastructure
Building an IoT SuperNetwork on top of the AWS Global InfrastructureBuilding an IoT SuperNetwork on top of the AWS Global Infrastructure
Building an IoT SuperNetwork on top of the AWS Global InfrastructureSteffen Gebert
 
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...Steffen Gebert
 
Feature Management Platforms
Feature Management PlatformsFeature Management Platforms
Feature Management PlatformsSteffen Gebert
 
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)Steffen Gebert
 
Jenkins vs. AWS CodePipeline
Jenkins vs. AWS CodePipelineJenkins vs. AWS CodePipeline
Jenkins vs. AWS CodePipelineSteffen Gebert
 
Monitoring Akka with Kamon 1.0
Monitoring Akka with Kamon 1.0Monitoring Akka with Kamon 1.0
Monitoring Akka with Kamon 1.0Steffen Gebert
 
(Declarative) Jenkins Pipelines
(Declarative) Jenkins Pipelines(Declarative) Jenkins Pipelines
(Declarative) Jenkins PipelinesSteffen Gebert
 
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins PipelinesAn Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins PipelinesSteffen Gebert
 
Let's go HTTPS-only! - More Than Buying a Certificate
Let's go HTTPS-only! - More Than Buying a CertificateLet's go HTTPS-only! - More Than Buying a Certificate
Let's go HTTPS-only! - More Than Buying a CertificateSteffen Gebert
 
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the Web
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the WebCleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the Web
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the WebSteffen Gebert
 
Investigating the Impact of Network Topology on the Processing Times of SDN C...
Investigating the Impact of Network Topology on the Processing Times of SDN C...Investigating the Impact of Network Topology on the Processing Times of SDN C...
Investigating the Impact of Network Topology on the Processing Times of SDN C...Steffen Gebert
 
SDN interfaces and performance analysis of SDN components
SDN interfaces and performance analysis of SDN componentsSDN interfaces and performance analysis of SDN components
SDN interfaces and performance analysis of SDN componentsSteffen Gebert
 
The Development Infrastructure of the TYPO3 Project
The Development Infrastructure of the TYPO3 ProjectThe Development Infrastructure of the TYPO3 Project
The Development Infrastructure of the TYPO3 ProjectSteffen Gebert
 
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-Entwicklung
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-EntwicklungDer Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-Entwicklung
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-EntwicklungSteffen Gebert
 
Official typo3.org infrastructure &
the TYPO3 Server Admin Team
Official typo3.org infrastructure &
the TYPO3 Server Admin TeamOfficial typo3.org infrastructure &
the TYPO3 Server Admin Team
Official typo3.org infrastructure &
the TYPO3 Server Admin TeamSteffen Gebert
 
Neuigkeiten aus dem TYPO3-Projekt
Neuigkeiten aus dem TYPO3-ProjektNeuigkeiten aus dem TYPO3-Projekt
Neuigkeiten aus dem TYPO3-ProjektSteffen Gebert
 
The TYPO3 Server Admin Team
The TYPO3 Server Admin TeamThe TYPO3 Server Admin Team
The TYPO3 Server Admin TeamSteffen Gebert
 

More from Steffen Gebert (20)

Building an IoT SuperNetwork on top of the AWS Global Infrastructure
Building an IoT SuperNetwork on top of the AWS Global InfrastructureBuilding an IoT SuperNetwork on top of the AWS Global Infrastructure
Building an IoT SuperNetwork on top of the AWS Global Infrastructure
 
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...
 
Feature Management Platforms
Feature Management PlatformsFeature Management Platforms
Feature Management Platforms
 
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
 
Jenkins vs. AWS CodePipeline
Jenkins vs. AWS CodePipelineJenkins vs. AWS CodePipeline
Jenkins vs. AWS CodePipeline
 
Monitoring Akka with Kamon 1.0
Monitoring Akka with Kamon 1.0Monitoring Akka with Kamon 1.0
Monitoring Akka with Kamon 1.0
 
(Declarative) Jenkins Pipelines
(Declarative) Jenkins Pipelines(Declarative) Jenkins Pipelines
(Declarative) Jenkins Pipelines
 
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins PipelinesAn Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
 
Continuous Delivery
Continuous DeliveryContinuous Delivery
Continuous Delivery
 
Jenkins Pipelines
Jenkins PipelinesJenkins Pipelines
Jenkins Pipelines
 
Let's go HTTPS-only! - More Than Buying a Certificate
Let's go HTTPS-only! - More Than Buying a CertificateLet's go HTTPS-only! - More Than Buying a Certificate
Let's go HTTPS-only! - More Than Buying a Certificate
 
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the Web
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the WebCleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the Web
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the Web
 
Investigating the Impact of Network Topology on the Processing Times of SDN C...
Investigating the Impact of Network Topology on the Processing Times of SDN C...Investigating the Impact of Network Topology on the Processing Times of SDN C...
Investigating the Impact of Network Topology on the Processing Times of SDN C...
 
SDN interfaces and performance analysis of SDN components
SDN interfaces and performance analysis of SDN componentsSDN interfaces and performance analysis of SDN components
SDN interfaces and performance analysis of SDN components
 
Git Power-Workshop
Git Power-WorkshopGit Power-Workshop
Git Power-Workshop
 
The Development Infrastructure of the TYPO3 Project
The Development Infrastructure of the TYPO3 ProjectThe Development Infrastructure of the TYPO3 Project
The Development Infrastructure of the TYPO3 Project
 
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-Entwicklung
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-EntwicklungDer Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-Entwicklung
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-Entwicklung
 
Official typo3.org infrastructure &
the TYPO3 Server Admin Team
Official typo3.org infrastructure &
the TYPO3 Server Admin TeamOfficial typo3.org infrastructure &
the TYPO3 Server Admin Team
Official typo3.org infrastructure &
the TYPO3 Server Admin Team
 
Neuigkeiten aus dem TYPO3-Projekt
Neuigkeiten aus dem TYPO3-ProjektNeuigkeiten aus dem TYPO3-Projekt
Neuigkeiten aus dem TYPO3-Projekt
 
The TYPO3 Server Admin Team
The TYPO3 Server Admin TeamThe TYPO3 Server Admin Team
The TYPO3 Server Admin Team
 

Recently uploaded

Women in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationWomen in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationDianaGray10
 
Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactivestartupro
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
Introduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxIntroduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxmprakaash5
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...BookNet Canada
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementDianaGray10
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 

Recently uploaded (20)

Women in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationWomen in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automation
 
Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactive
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
Introduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxIntroduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptx
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions management
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 

How our Cloudy Mindsets Approached Physical Routers

  • 1. How our Cloudy Mindsets Approached Physical Routers SNMP was not an option Steffen Gebert DENOG12, 09.11.2020
  • 2. After the latest project, EMnify became a 99% only cloud company. To meet growing scalability and reliability requirements of the interconnection between our AWS- based deployments and multiple carriers, BGP peerings had to be moved out of AWS. Therefore, a pair of Juniper routers were put into place. For a company fully relying on cloud services so far, this alien technology resulted in several challenges. We want to share, how we solved the integration puzzle of this physical equipment into our existing workflows and tools. The use of CI/CD systems for applying changes, AWS CloudWatch, Prometheus and Grafana for monitoring as well as the reluctance to run applications that require a lot of shepherding lead our research to find the right glue - the glue between these pieces of iron and our cloud infrastructure. Being used to CI/CD processes backed by automated tests, we wanted to adapt these practices here as well. As a result, configuration changes are rolled out by an automated pipeline using Ansible. Efforts for automated testing were made, where we failed. We explain why and what we did instead as well as what we envision for the future. As every other part of our system, we want its monitoring data accessible via Grafana. With the help of pmacct and fluentbit, we can treat IPFIX flow records as they were logs. With the help of jtimon, Prometheus stores the routers’ metrics as we are used to do, in doubt tickled out through few custom YANG models. In summary, the integration worked very well, while we still have several learnings and pain points to share. Abstract
  • 3. Thanks to our Sponsors!
  • 5. 5 years ago All rights reserved | Confidential 10. 11.
  • 8. Is This a Better World?
  • 13. EMnify’s IoT Connectivity Platform 13@StGebert Cellular connectivity in 500+ networks in 185 countries RESTful APIs Pay as you go pricing SMS/USSD to REST bridge Secure connectivity via VPN and AWS natively Implemented using own virtualized mobile core network
  • 14. Supporting Global IoT Deployments Home-routing of roaming SIM data prevents distributed architecture EMnify’s mobile core network is deployed in multiple AWS regions – keeping data local @StGebert 14
  • 15. GRX/IPX Network (GPRS Roaming Exchange) 15 Internet Telco world GRX/IPX @StGebert
  • 17. We’re critical to our customers’ success
  • 18. Increased demands vs. AWS as “General Purpose Cloud”
  • 20. We had to move logic out of AWS
  • 21. We could not find a fitting managed service
  • 22. We had to get hardware
  • 23. We chose boring technology
  • 25. • Juniper MX204 • Colocation rack space • Fiber links towards 2 carriers AWS • Out-of-band management access via OPNsense 25@StGebert Setup – Twice per region
  • 27. 27 Design Principles 80/20 rule aka MVP Don’t get out of our comfort zone Don’t setup anythat that requires lot of handholding @StGebert
  • 29. A human shall not SSH into something M Y I N N E R S E L F
  • 31. Configuration Deployment Render Config •Single config file •Parametrized using variables Commit •Upload using juniper_junos_config •Load override mode •Issue (to be) confirmed commit Test •Ping tests Confirm •Confirm commit @StGebert 31
  • 32. Ansible Playbook - Code Example @StGebert 32 - name: install generated configuration file onto device juniper_junos_config: provider: "{{ juniper_connection_settings }}" src: "{{ conf_file }}" load: override comment: "playbook execution, commit confirmed" confirmed: 3 # wait X minutes until rollback diff: yes ignore_warning: yes register: config_results notify: confirm previous commit
  • 33. • Separate AWS account • Isolated connectivity 33@StGebert Config Pipeline AWS CodePipeline AWS CodeBuild
  • 34. And where are the tests?
  • 35. In a Perfect World.. 35 • 2-star review could have been mine :D • Latest version: 18.4R1 • Takes ~30min to be ready • AWS does not support VLANs! • Only for manual testing • Maybe eve-ng or GNS3 could help? @StGebert
  • 36. • Start virtualized topology in network emulator • Apply configuration pipeline • Emulate BGP peers • Execute end-to-end connectivity tests • Emulate link failures • Verify connectivity • AWS: run on bare metal host (b/c CPU VMX) On My Bucket List 36@StGebert
  • 38. 38@StGebert Firmware Update - Checks /workspace/prod # ansible-playbook upgrade_check.yaml -u steffen.gebert ... TASK [Validate result] ***************** [ mx204-am3 ] Chassis Alarms ---------------------------- Expect: No alarms currently active Actual: No alarms currently active ... [ mx204-am3 ] Core Dumps ---------------------------- Expect: /var/crash/*core*: No such file or directory Actual: /var/crash/*core*: No such file or directory [ mx204-am3 ] ⚠ Proceed? ⚠ : Press 'C' to continue the play or 'A' to abort
  • 39. 39@StGebert Firmware Update - Draining - name: Drain traffic juniper_junos_config: provider: "{{ juniper_connection_settings }}" load: 'set' lines: - 'activate policy-options policy-statement OUT-OF-SERVICE-SWITCH term as-path-pr comment: 'Drain traffic to router for upgrade' - name: Traffic drained pause: prompt: | [ {{item}} ] Traffic is draining. Verify that traffic is completely drained on the following dashboard before proce [ {{item}} ] ⚠ Proceed with the JunOS upgrade ⚠? loop: "{{ ansible_play_hosts }}"
  • 40. 40@StGebert Firmware Update – Execute! - name: Install Junos OS package juniper_junos_software: provider: host: "{{ ansible_host }}" timeout: 3600 remote_package: "{{ junos_vm_file }}" validate: True cleanfs: False vmhost: True reboot: True ignore_errors: yes # rpc times out when upgrading, despite the provider timeout setti register: output
  • 41. Deploy a file ¯_(ツ)_/¯ Challenges Max length of file copy URLs Feedback for invalid confg Amount of boilerplate code
  • 44. •Who logged into the router? •What’s happening in the router? 44@StGebert Syslog Implementation rsyslog Amazon CloudWatch Logs 1 containers to run
  • 46. Flow Records Network-to- Network Interface (GTP traffic) ~20k parallel flows 1 flow => 1 log line?! 46@StGebert
  • 47. Amazon CloudWatch Logs •How much traffic per AS? •Did we receive any signaling from XYZ and did we really respond? 47@StGebert Flow Records Collection IPFIX 2 containers to run BMP FireLens
  • 48. pmacct / nfacct @StGebert 48 libpcap NFLOG NetFlow IPFIX BGP BMP RPKI tag filter aggregate split RDBMS NoSQL MQ NetFlow Memory File stdout
  • 49. protocol Enrichment @StGebert 49 host_ip src_ip dst_ip src_port dst_port src_asn dst_asn src_ifindex dst_ifindex … direction organization hostname carrier drop BGP & BFD
  • 50. Lua Magic @StGebert 50 -- Sets GTP-c or GTP-u protocol depending on port numbers function setGTPProtocol(tag, timestamp, record) local code = 0 local gtp_ports = { ["GTP-c"] = 2123, ["GTP-u"] = 2152, ["GTP'"] = 3386, } local new_record = record for protocol, port in pairs(gtp_ports) do if record["source.port"] == port or record["destination.port"] == port then new_record["network.application"] = "GTP" new_record["network.protocol"] = protocol code = 2 end end return code, timestamp, new_record
  • 53. Inbound traffic by AS query @StGebert 53 fields concat(source.as.organization.name, ', ’, source.as.organization.country, ' (AS ', source.as.number, ')') as org | filter @logStream = "flows" | filter host.name like /^$host$/ | filter concat(source.as.number, ' ', source.as.organization.name, ' ', source.as.organization.country,' ',source.as.organization.tadig) like /$operator/ OR concat(destination.as.number, ' ', destination.as.organization.name, ' ’, destination.as.organization.country, ' ', destination.as.organization.tadig) like /$operator/ | filter network.peer.destination.as.organization.name like /^$carrier$/ | filter network.direction = "inbound" | filter network.protocol like /$protocol/ | filter 10000 | stats sum(network.bytes)/60*8 as `` by org,bin($time_interval) | sort `` desc
  • 56. Metrics Demand 56 Temperature, light levels, etc. State, throughput, errors, etc. State, prefixes received/accepted/installed Hard- ware Inter- faces BGP @StGebert
  • 57. Prometheus • High cardinality, high frequency metrics collection 57@StGebert Metrics Implementation Junos Telemetry (JTI) 1 container to run JTIMON
  • 61. Prometheus • Prometheus-integrated alerting 61@StGebert Alerting Implementation Junos Telemetry (JTI) JTIMON Alertmanager
  • 62. • Integrated hardware into an otherwise fully cloud-based environment • Avoid new processes • Avoid new (user-facing) tooling • Found tooling to bridge gaps to “what we’re comfortable with” • 1-2 containers running existing open source tooling • No guarantuee that this scales to 10s of devices • Please contact me, if you want details (configs etc.) or have suggestions! 62@StGebert Summary & Conclusion