SlideShare a Scribd company logo
FFMUC: Half a year with WireGuard
VXLAN + B.A.T.M.A.N. and some python included
FFWCW 2021
awlnx
● Annika Wickert
● Senior Network Engineer / OpenSource since 2010
● Twitter @awlnx / Github @awlx
2
Who am I?
3
FFMUC?
• Freie Netze München e.V. since 2014
• Community Freifunk München since 2004
• Wifi
• #FFMEET
• DoH/DoT/DNSCrypt/DNS
• Streaming
4
FFMUC ran on fastd
• FFMUC was built with fastd and B.A.T.M.A.N.
• We got bigger compute nodes and bigger uplinks - we wanted to leverage the
resources
• We didn’t want to change too much at once => not too much risk
• So why not change _only_ the transport network and keep B.A.T.M.A.N.
5
Wireguard vs fastd
• Fastd is a single threaded userspace process
• WireGuard runs in kernel space thus has to be multi threaded
• WireGuard cannot transport Layer 2 protocols - B.A.T.M.A.N. is one ...
• We need another encapsulation which solves this problem => VXLAN
Wireguard
VXLAN
B.A.T.M.A.N.
6
What does it look like in the end?
7
Challenges we already knew
• No systemd-networkd support for B.A.T.M.A.N.
• We are an open network - we don’t want node owners to signup
• WireGuard has a pre-shared key infra
=> we need a daemon which handles incoming keys and programs them
to the gateways
8
WGKex!
9
How does it work?
• WireGuard peers on the gateways are created by wgkex
• Allowed IP is derived from the public key of the node
• VxLAN Forwarding database entries are created by wgkex
10
Get in touch with maintainers
• To get validation data correct for wgkex etc
• We contacted WireGuard maintainers early in the process
• Asked questions about known scaling issues
• Opened PRs early as drafts to see if there is a chance of merging
• systemd-networkd https://github.com/systemd/systemd/pull/17252
• gluon-community-packages
https://github.com/freifunk-gluon/community-packages/pull/6
11
Solve problems upstream!
• We invested much time in systemd-networkd
• We wanted to get our stuff merged in upstream
• No custom solutions for our setup, just upstream compatible which solves many
resource problems in the future
12
Gateways
• Everything is automated with Saltstack
• systemd-networkd takes care of all interfaces
• 800 - 1000 Nodes per gateway are easy
• We are able to run whole FFMUC on just two gateways
13
Debugging … Flamegraphs and Bugs
• WireGuard performs well but we have too much load on our gateways. Why?
14
Upstream fixes!
• B.A.T.M.A.N.
■ https://patchwork.open-mesh.org/project/b.a.t.m.a.n./patch/20201126
153120.1053700-1-sven@narfation.org/
■ https://patchwork.open-mesh.org/project/b.a.t.m.a.n./patch/20201127
173849.19208-4-sw@simonwunderlich.de/
■ https://patchwork.open-mesh.org/project/b.a.t.m.a.n./patch/20201127
173849.19208-2-sw@simonwunderlich.de/
• VxLAN
■ https://patchwork.open-mesh.org/project/b.a.t.m.a.n./patch/20201126
125247.1047977-1-sven@narfation.org/
15
Keep your NTP sync!
• Sync NTP before you try to connect to WireGuard
• If you don’t do that many funky things happen
• OpenWRT defaults its clock to build date of firmware so it works the first few
days after release … because it’s good enough
16
Not enough random during boot
• ERX didn’t have a good enough random seed …
• After flashing, it’s unreachable for … hours … days … maybe weeks?
=> fixed
https://github.com/oszilloskop/UBNT_ERX_Gluon_Factory-Image/issues/
3
17
So is it faster?
18
Lessons learned
• Commit as much stuff as possible upstream
• Work close with upstream
• Get much feedback from all the communities/other people
• Involve as many people as you can
• Start your project anyway ;)
19
What’s next?
• We want to get rid of B.A.T.M.A.N. for gateway uplinks (make broadcast
domains small)
■ Should boost performance by 5x to 7x depending on CPU
■ Maybe VxLAN first, then a fully routed approach
■ https://github.com/freifunkMUC/site-ffm/issues/87
20
Community
• Freifunk Darmstadt and Freifunk Regensburg helped a lot during development
of wgkex!
• B.A.T.M.A.N. developers helped a lot during debugging the performance issue
and created many bugfixes
• Everything is opensource and available on Github
https://github.com/freifunkMUC
• More background and all fixes:
https://ffmuc.net/freifunkmuc/2020/12/03/wireguard-firmware/
21
Thanks to everyone involved
• Freifunk Darmstadt @hexa
• Freifunk Regensburg @MoepMan
• Freifunk Hannover @aiyion, @Codefetch
• systemd Yu Watanabe, Lennart Poettering
• WireGuard Jason A. Donenfeld
• B.A.T.M.A.N. @ecsv @T_X
• All the folks of FFMUC for testing
• Everyone else who we forgot and was involved in any way
=> Community rocks! #Together #OpenSource

More Related Content

What's hot

Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014
Netgate
 
Vagrant + SaltStack + Django - Ararat Poghosyan - DM10
Vagrant + SaltStack + Django - Ararat Poghosyan - DM10Vagrant + SaltStack + Django - Ararat Poghosyan - DM10
Vagrant + SaltStack + Django - Ararat Poghosyan - DM10
Ararat Poghosyan
 
Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014
Netgate
 
Janus/HOMER/HEPIC @ OpenSIPS18
Janus/HOMER/HEPIC @ OpenSIPS18Janus/HOMER/HEPIC @ OpenSIPS18
Janus/HOMER/HEPIC @ OpenSIPS18
Lorenzo Miniero
 
Fixing Docker networking - Milos Gajdos at #DOXLON
Fixing Docker networking - Milos Gajdos at #DOXLONFixing Docker networking - Milos Gajdos at #DOXLON
Fixing Docker networking - Milos Gajdos at #DOXLON
Outlyer
 
Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014
Netgate
 
Building a Small DC
Building a Small DCBuilding a Small DC
Building a Small DC
APNIC
 
Shiny New HTTP Shit
Shiny New HTTP ShitShiny New HTTP Shit
Shiny New HTTP Shit
Mark Nottingham
 
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - KingOpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebula Project
 
Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019
Lorenzo Miniero
 
Welcome talk unleashing the future of open-source enterprise cloud computing
Welcome talk   unleashing the future of open-source enterprise cloud computingWelcome talk   unleashing the future of open-source enterprise cloud computing
Welcome talk unleashing the future of open-source enterprise cloud computing
NETWAYS
 
Home Automation
Home AutomationHome Automation
Home Automation
Cássio Landim
 
OAuth and STUN, TURN in WebRTC context RFC7635
OAuth and STUN, TURN  in WebRTC context RFC7635OAuth and STUN, TURN  in WebRTC context RFC7635
OAuth and STUN, TURN in WebRTC context RFC7635
Mihály Mészáros
 
Kamailio presence + json
Kamailio presence + jsonKamailio presence + json
Kamailio presence + json
Emmanuel Schmidbauer
 
OSMC 2013 | Zabbix: A Practical Demo by Rihards Olups
OSMC 2013 | Zabbix: A Practical Demo by Rihards OlupsOSMC 2013 | Zabbix: A Practical Demo by Rihards Olups
OSMC 2013 | Zabbix: A Practical Demo by Rihards Olups
NETWAYS
 
Http/2 lightning
Http/2   lightningHttp/2   lightning
Http/2 lightning
Adrian Cardenas
 
Fuzzing Janus @ IPTComm 2019
Fuzzing Janus @ IPTComm 2019Fuzzing Janus @ IPTComm 2019
Fuzzing Janus @ IPTComm 2019
Lorenzo Miniero
 
Communicating on the web
Communicating on the webCommunicating on the web
Communicating on the web
Adrian Cardenas
 
IoT Project postmortem
IoT Project postmortemIoT Project postmortem
IoT Project postmortem
Matheus Marabesi
 
SPDY and What to Consider for HTTP/2.0
SPDY and What to Consider for HTTP/2.0SPDY and What to Consider for HTTP/2.0
SPDY and What to Consider for HTTP/2.0
Mike Belshe
 

What's hot (20)

Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014
 
Vagrant + SaltStack + Django - Ararat Poghosyan - DM10
Vagrant + SaltStack + Django - Ararat Poghosyan - DM10Vagrant + SaltStack + Django - Ararat Poghosyan - DM10
Vagrant + SaltStack + Django - Ararat Poghosyan - DM10
 
Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014
 
Janus/HOMER/HEPIC @ OpenSIPS18
Janus/HOMER/HEPIC @ OpenSIPS18Janus/HOMER/HEPIC @ OpenSIPS18
Janus/HOMER/HEPIC @ OpenSIPS18
 
Fixing Docker networking - Milos Gajdos at #DOXLON
Fixing Docker networking - Milos Gajdos at #DOXLONFixing Docker networking - Milos Gajdos at #DOXLON
Fixing Docker networking - Milos Gajdos at #DOXLON
 
Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014
 
Building a Small DC
Building a Small DCBuilding a Small DC
Building a Small DC
 
Shiny New HTTP Shit
Shiny New HTTP ShitShiny New HTTP Shit
Shiny New HTTP Shit
 
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - KingOpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
 
Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019
 
Welcome talk unleashing the future of open-source enterprise cloud computing
Welcome talk   unleashing the future of open-source enterprise cloud computingWelcome talk   unleashing the future of open-source enterprise cloud computing
Welcome talk unleashing the future of open-source enterprise cloud computing
 
Home Automation
Home AutomationHome Automation
Home Automation
 
OAuth and STUN, TURN in WebRTC context RFC7635
OAuth and STUN, TURN  in WebRTC context RFC7635OAuth and STUN, TURN  in WebRTC context RFC7635
OAuth and STUN, TURN in WebRTC context RFC7635
 
Kamailio presence + json
Kamailio presence + jsonKamailio presence + json
Kamailio presence + json
 
OSMC 2013 | Zabbix: A Practical Demo by Rihards Olups
OSMC 2013 | Zabbix: A Practical Demo by Rihards OlupsOSMC 2013 | Zabbix: A Practical Demo by Rihards Olups
OSMC 2013 | Zabbix: A Practical Demo by Rihards Olups
 
Http/2 lightning
Http/2   lightningHttp/2   lightning
Http/2 lightning
 
Fuzzing Janus @ IPTComm 2019
Fuzzing Janus @ IPTComm 2019Fuzzing Janus @ IPTComm 2019
Fuzzing Janus @ IPTComm 2019
 
Communicating on the web
Communicating on the webCommunicating on the web
Communicating on the web
 
IoT Project postmortem
IoT Project postmortemIoT Project postmortem
IoT Project postmortem
 
SPDY and What to Consider for HTTP/2.0
SPDY and What to Consider for HTTP/2.0SPDY and What to Consider for HTTP/2.0
SPDY and What to Consider for HTTP/2.0
 

Similar to FFMUC: Half a year with WireGuard

Tech Tutorial by Vikram Dham: Let's build MPLS router using SDN
Tech Tutorial by Vikram Dham: Let's build MPLS router using SDNTech Tutorial by Vikram Dham: Let's build MPLS router using SDN
Tech Tutorial by Vikram Dham: Let's build MPLS router using SDN
nvirters
 
How we use Twisted in Launchpad
How we use Twisted in LaunchpadHow we use Twisted in Launchpad
How we use Twisted in Launchpad
Michael Hudson-Doyle
 
Realtime traffic analyser
Realtime traffic analyserRealtime traffic analyser
Realtime traffic analyser
Alex Moskvin
 
Network troubleshooting
Network troubleshootingNetwork troubleshooting
Network troubleshooting
Skillspire LLC
 
Three years of OFELIA - taking stock
Three years of OFELIA - taking stockThree years of OFELIA - taking stock
Three years of OFELIA - taking stock
FIBRE Testbed
 
Balázs Bucsay - XFLTReaT: Building a Tunnel
Balázs Bucsay - XFLTReaT: Building a TunnelBalázs Bucsay - XFLTReaT: Building a Tunnel
Balázs Bucsay - XFLTReaT: Building a Tunnel
hacktivity
 
Introducing Container Technology to TSUBAME3.0 Supercomputer
Introducing Container Technology to TSUBAME3.0 SupercomputerIntroducing Container Technology to TSUBAME3.0 Supercomputer
Introducing Container Technology to TSUBAME3.0 Supercomputer
Akihiro Nomura
 
Docker for Mac and Windows: The Insider's Guide by Justin Cormack
Docker for Mac and Windows: The Insider's Guide by Justin CormackDocker for Mac and Windows: The Insider's Guide by Justin Cormack
Docker for Mac and Windows: The Insider's Guide by Justin Cormack
Docker, Inc.
 
Advanced Internet of Things firmware engineering with Thingsquare and Contiki...
Advanced Internet of Things firmware engineering with Thingsquare and Contiki...Advanced Internet of Things firmware engineering with Thingsquare and Contiki...
Advanced Internet of Things firmware engineering with Thingsquare and Contiki...
Adam Dunkels
 
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
 
Netflix Open Source: Building a Distributed and Automated Open Source Program
Netflix Open Source:  Building a Distributed and Automated Open Source ProgramNetflix Open Source:  Building a Distributed and Automated Open Source Program
Netflix Open Source: Building a Distributed and Automated Open Source Program
aspyker
 
Building a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at NetflixBuilding a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at Netflix
All Things Open
 
The internet of $h1t
The internet of $h1tThe internet of $h1t
The internet of $h1t
Amit Serper
 
Monkey Server
Monkey ServerMonkey Server
Monkey Server
Eduardo Silva Pereira
 
Docker and Fluentd
Docker and FluentdDocker and Fluentd
Docker and Fluentd
N Masahiro
 
SDN Presentation
SDN PresentationSDN Presentation
SDN Presentation
Abderrahmane TEKFI
 
SDN Demystified, by Dean Pemberton [APNIC 38]
SDN Demystified, by Dean Pemberton [APNIC 38]SDN Demystified, by Dean Pemberton [APNIC 38]
SDN Demystified, by Dean Pemberton [APNIC 38]
APNIC
 
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernelUnikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
The Linux Foundation
 
Building a Small Datacenter
Building a Small DatacenterBuilding a Small Datacenter
Building a Small Datacenter
ssuser4b98f0
 
Open Source Investments in Mainframe Through the Next Generation - Showcasing...
Open Source Investments in Mainframe Through the Next Generation - Showcasing...Open Source Investments in Mainframe Through the Next Generation - Showcasing...
Open Source Investments in Mainframe Through the Next Generation - Showcasing...
Open Mainframe Project
 

Similar to FFMUC: Half a year with WireGuard (20)

Tech Tutorial by Vikram Dham: Let's build MPLS router using SDN
Tech Tutorial by Vikram Dham: Let's build MPLS router using SDNTech Tutorial by Vikram Dham: Let's build MPLS router using SDN
Tech Tutorial by Vikram Dham: Let's build MPLS router using SDN
 
How we use Twisted in Launchpad
How we use Twisted in LaunchpadHow we use Twisted in Launchpad
How we use Twisted in Launchpad
 
Realtime traffic analyser
Realtime traffic analyserRealtime traffic analyser
Realtime traffic analyser
 
Network troubleshooting
Network troubleshootingNetwork troubleshooting
Network troubleshooting
 
Three years of OFELIA - taking stock
Three years of OFELIA - taking stockThree years of OFELIA - taking stock
Three years of OFELIA - taking stock
 
Balázs Bucsay - XFLTReaT: Building a Tunnel
Balázs Bucsay - XFLTReaT: Building a TunnelBalázs Bucsay - XFLTReaT: Building a Tunnel
Balázs Bucsay - XFLTReaT: Building a Tunnel
 
Introducing Container Technology to TSUBAME3.0 Supercomputer
Introducing Container Technology to TSUBAME3.0 SupercomputerIntroducing Container Technology to TSUBAME3.0 Supercomputer
Introducing Container Technology to TSUBAME3.0 Supercomputer
 
Docker for Mac and Windows: The Insider's Guide by Justin Cormack
Docker for Mac and Windows: The Insider's Guide by Justin CormackDocker for Mac and Windows: The Insider's Guide by Justin Cormack
Docker for Mac and Windows: The Insider's Guide by Justin Cormack
 
Advanced Internet of Things firmware engineering with Thingsquare and Contiki...
Advanced Internet of Things firmware engineering with Thingsquare and Contiki...Advanced Internet of Things firmware engineering with Thingsquare and Contiki...
Advanced Internet of Things firmware engineering with Thingsquare and Contiki...
 
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
 
Netflix Open Source: Building a Distributed and Automated Open Source Program
Netflix Open Source:  Building a Distributed and Automated Open Source ProgramNetflix Open Source:  Building a Distributed and Automated Open Source Program
Netflix Open Source: Building a Distributed and Automated Open Source Program
 
Building a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at NetflixBuilding a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at Netflix
 
The internet of $h1t
The internet of $h1tThe internet of $h1t
The internet of $h1t
 
Monkey Server
Monkey ServerMonkey Server
Monkey Server
 
Docker and Fluentd
Docker and FluentdDocker and Fluentd
Docker and Fluentd
 
SDN Presentation
SDN PresentationSDN Presentation
SDN Presentation
 
SDN Demystified, by Dean Pemberton [APNIC 38]
SDN Demystified, by Dean Pemberton [APNIC 38]SDN Demystified, by Dean Pemberton [APNIC 38]
SDN Demystified, by Dean Pemberton [APNIC 38]
 
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernelUnikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
 
Building a Small Datacenter
Building a Small DatacenterBuilding a Small Datacenter
Building a Small Datacenter
 
Open Source Investments in Mainframe Through the Next Generation - Showcasing...
Open Source Investments in Mainframe Through the Next Generation - Showcasing...Open Source Investments in Mainframe Through the Next Generation - Showcasing...
Open Source Investments in Mainframe Through the Next Generation - Showcasing...
 

Recently uploaded

Bitcoin vs Ethereum Which Crypto Performed Better in Q2, 2024.docx
Bitcoin vs Ethereum Which Crypto Performed Better in Q2, 2024.docxBitcoin vs Ethereum Which Crypto Performed Better in Q2, 2024.docx
Bitcoin vs Ethereum Which Crypto Performed Better in Q2, 2024.docx
SFC Today
 
Best CSS Animation Libraries for Web Developers
Best CSS Animation Libraries for Web DevelopersBest CSS Animation Libraries for Web Developers
Best CSS Animation Libraries for Web Developers
Shrestha Raaz
 
202254.com香蕉影视,在线观看《我才不要和你做朋友呢》在线观看最新电影,香蕉影视在线观看《我才不要和你做朋友呢》在线观看高清电影
202254.com香蕉影视,在线观看《我才不要和你做朋友呢》在线观看最新电影,香蕉影视在线观看《我才不要和你做朋友呢》在线观看高清电影202254.com香蕉影视,在线观看《我才不要和你做朋友呢》在线观看最新电影,香蕉影视在线观看《我才不要和你做朋友呢》在线观看高清电影
202254.com香蕉影视,在线观看《我才不要和你做朋友呢》在线观看最新电影,香蕉影视在线观看《我才不要和你做朋友呢》在线观看高清电影
ffg01100
 
Trump Assassination Shirt Trump Assassination Shirt
Trump Assassination Shirt Trump Assassination ShirtTrump Assassination Shirt Trump Assassination Shirt
Trump Assassination Shirt Trump Assassination Shirt
exgf28
 
Week 1 - Pendidikan Pancasila - Gr 1.docx
Week 1 - Pendidikan Pancasila - Gr 1.docxWeek 1 - Pendidikan Pancasila - Gr 1.docx
Week 1 - Pendidikan Pancasila - Gr 1.docx
JunaManroe1
 
High Profile Girls Call ServiCe Chennai XX00XXX00X Tanisha Best High Class Ch...
High Profile Girls Call ServiCe Chennai XX00XXX00X Tanisha Best High Class Ch...High Profile Girls Call ServiCe Chennai XX00XXX00X Tanisha Best High Class Ch...
High Profile Girls Call ServiCe Chennai XX00XXX00X Tanisha Best High Class Ch...
shamrisumri
 
Use of Ontologies in Chemical Kinetic Database CHEMCONNECT
Use of Ontologies in Chemical Kinetic Database CHEMCONNECTUse of Ontologies in Chemical Kinetic Database CHEMCONNECT
Use of Ontologies in Chemical Kinetic Database CHEMCONNECT
Edward Blurock
 
Portugal Dreamin 24 - How to easily use an API with Flows
Portugal Dreamin 24  - How to easily use an API with FlowsPortugal Dreamin 24  - How to easily use an API with Flows
Portugal Dreamin 24 - How to easily use an API with Flows
Thierry TROUIN ☁
 
Effective Tips for Creating the Best Rich Media Ads .pptx
Effective Tips for Creating the Best Rich Media Ads .pptxEffective Tips for Creating the Best Rich Media Ads .pptx
Effective Tips for Creating the Best Rich Media Ads .pptx
AirtoryInc
 
SisAi World - Software is AI - Providing AI as Software - Protecting the Inte...
SisAi World - Software is AI - Providing AI as Software - Protecting the Inte...SisAi World - Software is AI - Providing AI as Software - Protecting the Inte...
SisAi World - Software is AI - Providing AI as Software - Protecting the Inte...
QingjieDu1
 
Ontology for the semantic enhancement, database definition and management and...
Ontology for the semantic enhancement, database definition and management and...Ontology for the semantic enhancement, database definition and management and...
Ontology for the semantic enhancement, database definition and management and...
Edward Blurock
 
How God led me to DTS? Through many different signs and connections that I c...
How God led me to DTS? Through many different signs and connections that  I c...How God led me to DTS? Through many different signs and connections that  I c...
How God led me to DTS? Through many different signs and connections that I c...
AshishMohan57
 
Chennai Girls Call ServiCe X00XXX00XX Tanisha Best High Class Chennai Available
Chennai Girls Call ServiCe X00XXX00XX Tanisha Best High Class Chennai AvailableChennai Girls Call ServiCe X00XXX00XX Tanisha Best High Class Chennai Available
Chennai Girls Call ServiCe X00XXX00XX Tanisha Best High Class Chennai Available
shamrisumri
 
How-to-Diagnose-Hard-Drives-by-DFL-DDP-2024.pdf
How-to-Diagnose-Hard-Drives-by-DFL-DDP-2024.pdfHow-to-Diagnose-Hard-Drives-by-DFL-DDP-2024.pdf
How-to-Diagnose-Hard-Drives-by-DFL-DDP-2024.pdf
Dolphin Data Lab
 
Enhancing seamless access using TIGERfed
Enhancing seamless access using TIGERfedEnhancing seamless access using TIGERfed
Enhancing seamless access using TIGERfed
Bangladesh Network Operators Group
 
IPv6 Deployment Planning and Security Considerations
IPv6 Deployment Planning and Security ConsiderationsIPv6 Deployment Planning and Security Considerations
IPv6 Deployment Planning and Security Considerations
Bangladesh Network Operators Group
 
Digital ethnography of the Polish darknet drug trade community
Digital ethnography of the Polish darknet drug trade communityDigital ethnography of the Polish darknet drug trade community
Digital ethnography of the Polish darknet drug trade community
Piotr Siuda
 
Rent remote desktop server mangohost .net
Rent remote desktop server mangohost .netRent remote desktop server mangohost .net
Rent remote desktop server mangohost .net
pdfsubmission50
 
Why Your Business Needs a Professional Web Design Company UAE
Why Your Business Needs a Professional Web Design Company UAEWhy Your Business Needs a Professional Web Design Company UAE
Why Your Business Needs a Professional Web Design Company UAE
adelewhite125
 
Girls Call Mahipalpur 000XX00000 Provide Best And Top Girl Service And No1 in...
Girls Call Mahipalpur 000XX00000 Provide Best And Top Girl Service And No1 in...Girls Call Mahipalpur 000XX00000 Provide Best And Top Girl Service And No1 in...
Girls Call Mahipalpur 000XX00000 Provide Best And Top Girl Service And No1 in...
mahigarg2024#G05
 

Recently uploaded (20)

Bitcoin vs Ethereum Which Crypto Performed Better in Q2, 2024.docx
Bitcoin vs Ethereum Which Crypto Performed Better in Q2, 2024.docxBitcoin vs Ethereum Which Crypto Performed Better in Q2, 2024.docx
Bitcoin vs Ethereum Which Crypto Performed Better in Q2, 2024.docx
 
Best CSS Animation Libraries for Web Developers
Best CSS Animation Libraries for Web DevelopersBest CSS Animation Libraries for Web Developers
Best CSS Animation Libraries for Web Developers
 
202254.com香蕉影视,在线观看《我才不要和你做朋友呢》在线观看最新电影,香蕉影视在线观看《我才不要和你做朋友呢》在线观看高清电影
202254.com香蕉影视,在线观看《我才不要和你做朋友呢》在线观看最新电影,香蕉影视在线观看《我才不要和你做朋友呢》在线观看高清电影202254.com香蕉影视,在线观看《我才不要和你做朋友呢》在线观看最新电影,香蕉影视在线观看《我才不要和你做朋友呢》在线观看高清电影
202254.com香蕉影视,在线观看《我才不要和你做朋友呢》在线观看最新电影,香蕉影视在线观看《我才不要和你做朋友呢》在线观看高清电影
 
Trump Assassination Shirt Trump Assassination Shirt
Trump Assassination Shirt Trump Assassination ShirtTrump Assassination Shirt Trump Assassination Shirt
Trump Assassination Shirt Trump Assassination Shirt
 
Week 1 - Pendidikan Pancasila - Gr 1.docx
Week 1 - Pendidikan Pancasila - Gr 1.docxWeek 1 - Pendidikan Pancasila - Gr 1.docx
Week 1 - Pendidikan Pancasila - Gr 1.docx
 
High Profile Girls Call ServiCe Chennai XX00XXX00X Tanisha Best High Class Ch...
High Profile Girls Call ServiCe Chennai XX00XXX00X Tanisha Best High Class Ch...High Profile Girls Call ServiCe Chennai XX00XXX00X Tanisha Best High Class Ch...
High Profile Girls Call ServiCe Chennai XX00XXX00X Tanisha Best High Class Ch...
 
Use of Ontologies in Chemical Kinetic Database CHEMCONNECT
Use of Ontologies in Chemical Kinetic Database CHEMCONNECTUse of Ontologies in Chemical Kinetic Database CHEMCONNECT
Use of Ontologies in Chemical Kinetic Database CHEMCONNECT
 
Portugal Dreamin 24 - How to easily use an API with Flows
Portugal Dreamin 24  - How to easily use an API with FlowsPortugal Dreamin 24  - How to easily use an API with Flows
Portugal Dreamin 24 - How to easily use an API with Flows
 
Effective Tips for Creating the Best Rich Media Ads .pptx
Effective Tips for Creating the Best Rich Media Ads .pptxEffective Tips for Creating the Best Rich Media Ads .pptx
Effective Tips for Creating the Best Rich Media Ads .pptx
 
SisAi World - Software is AI - Providing AI as Software - Protecting the Inte...
SisAi World - Software is AI - Providing AI as Software - Protecting the Inte...SisAi World - Software is AI - Providing AI as Software - Protecting the Inte...
SisAi World - Software is AI - Providing AI as Software - Protecting the Inte...
 
Ontology for the semantic enhancement, database definition and management and...
Ontology for the semantic enhancement, database definition and management and...Ontology for the semantic enhancement, database definition and management and...
Ontology for the semantic enhancement, database definition and management and...
 
How God led me to DTS? Through many different signs and connections that I c...
How God led me to DTS? Through many different signs and connections that  I c...How God led me to DTS? Through many different signs and connections that  I c...
How God led me to DTS? Through many different signs and connections that I c...
 
Chennai Girls Call ServiCe X00XXX00XX Tanisha Best High Class Chennai Available
Chennai Girls Call ServiCe X00XXX00XX Tanisha Best High Class Chennai AvailableChennai Girls Call ServiCe X00XXX00XX Tanisha Best High Class Chennai Available
Chennai Girls Call ServiCe X00XXX00XX Tanisha Best High Class Chennai Available
 
How-to-Diagnose-Hard-Drives-by-DFL-DDP-2024.pdf
How-to-Diagnose-Hard-Drives-by-DFL-DDP-2024.pdfHow-to-Diagnose-Hard-Drives-by-DFL-DDP-2024.pdf
How-to-Diagnose-Hard-Drives-by-DFL-DDP-2024.pdf
 
Enhancing seamless access using TIGERfed
Enhancing seamless access using TIGERfedEnhancing seamless access using TIGERfed
Enhancing seamless access using TIGERfed
 
IPv6 Deployment Planning and Security Considerations
IPv6 Deployment Planning and Security ConsiderationsIPv6 Deployment Planning and Security Considerations
IPv6 Deployment Planning and Security Considerations
 
Digital ethnography of the Polish darknet drug trade community
Digital ethnography of the Polish darknet drug trade communityDigital ethnography of the Polish darknet drug trade community
Digital ethnography of the Polish darknet drug trade community
 
Rent remote desktop server mangohost .net
Rent remote desktop server mangohost .netRent remote desktop server mangohost .net
Rent remote desktop server mangohost .net
 
Why Your Business Needs a Professional Web Design Company UAE
Why Your Business Needs a Professional Web Design Company UAEWhy Your Business Needs a Professional Web Design Company UAE
Why Your Business Needs a Professional Web Design Company UAE
 
Girls Call Mahipalpur 000XX00000 Provide Best And Top Girl Service And No1 in...
Girls Call Mahipalpur 000XX00000 Provide Best And Top Girl Service And No1 in...Girls Call Mahipalpur 000XX00000 Provide Best And Top Girl Service And No1 in...
Girls Call Mahipalpur 000XX00000 Provide Best And Top Girl Service And No1 in...
 

FFMUC: Half a year with WireGuard

  • 1. FFMUC: Half a year with WireGuard VXLAN + B.A.T.M.A.N. and some python included FFWCW 2021
  • 2. awlnx ● Annika Wickert ● Senior Network Engineer / OpenSource since 2010 ● Twitter @awlnx / Github @awlx 2 Who am I?
  • 3. 3 FFMUC? • Freie Netze München e.V. since 2014 • Community Freifunk München since 2004 • Wifi • #FFMEET • DoH/DoT/DNSCrypt/DNS • Streaming
  • 4. 4 FFMUC ran on fastd • FFMUC was built with fastd and B.A.T.M.A.N. • We got bigger compute nodes and bigger uplinks - we wanted to leverage the resources • We didn’t want to change too much at once => not too much risk • So why not change _only_ the transport network and keep B.A.T.M.A.N.
  • 5. 5 Wireguard vs fastd • Fastd is a single threaded userspace process • WireGuard runs in kernel space thus has to be multi threaded • WireGuard cannot transport Layer 2 protocols - B.A.T.M.A.N. is one ... • We need another encapsulation which solves this problem => VXLAN Wireguard VXLAN B.A.T.M.A.N.
  • 6. 6 What does it look like in the end?
  • 7. 7 Challenges we already knew • No systemd-networkd support for B.A.T.M.A.N. • We are an open network - we don’t want node owners to signup • WireGuard has a pre-shared key infra => we need a daemon which handles incoming keys and programs them to the gateways
  • 9. 9 How does it work? • WireGuard peers on the gateways are created by wgkex • Allowed IP is derived from the public key of the node • VxLAN Forwarding database entries are created by wgkex
  • 10. 10 Get in touch with maintainers • To get validation data correct for wgkex etc • We contacted WireGuard maintainers early in the process • Asked questions about known scaling issues • Opened PRs early as drafts to see if there is a chance of merging • systemd-networkd https://github.com/systemd/systemd/pull/17252 • gluon-community-packages https://github.com/freifunk-gluon/community-packages/pull/6
  • 11. 11 Solve problems upstream! • We invested much time in systemd-networkd • We wanted to get our stuff merged in upstream • No custom solutions for our setup, just upstream compatible which solves many resource problems in the future
  • 12. 12 Gateways • Everything is automated with Saltstack • systemd-networkd takes care of all interfaces • 800 - 1000 Nodes per gateway are easy • We are able to run whole FFMUC on just two gateways
  • 13. 13 Debugging … Flamegraphs and Bugs • WireGuard performs well but we have too much load on our gateways. Why?
  • 14. 14 Upstream fixes! • B.A.T.M.A.N. ■ https://patchwork.open-mesh.org/project/b.a.t.m.a.n./patch/20201126 153120.1053700-1-sven@narfation.org/ ■ https://patchwork.open-mesh.org/project/b.a.t.m.a.n./patch/20201127 173849.19208-4-sw@simonwunderlich.de/ ■ https://patchwork.open-mesh.org/project/b.a.t.m.a.n./patch/20201127 173849.19208-2-sw@simonwunderlich.de/ • VxLAN ■ https://patchwork.open-mesh.org/project/b.a.t.m.a.n./patch/20201126 125247.1047977-1-sven@narfation.org/
  • 15. 15 Keep your NTP sync! • Sync NTP before you try to connect to WireGuard • If you don’t do that many funky things happen • OpenWRT defaults its clock to build date of firmware so it works the first few days after release … because it’s good enough
  • 16. 16 Not enough random during boot • ERX didn’t have a good enough random seed … • After flashing, it’s unreachable for … hours … days … maybe weeks? => fixed https://github.com/oszilloskop/UBNT_ERX_Gluon_Factory-Image/issues/ 3
  • 17. 17 So is it faster?
  • 18. 18 Lessons learned • Commit as much stuff as possible upstream • Work close with upstream • Get much feedback from all the communities/other people • Involve as many people as you can • Start your project anyway ;)
  • 19. 19 What’s next? • We want to get rid of B.A.T.M.A.N. for gateway uplinks (make broadcast domains small) ■ Should boost performance by 5x to 7x depending on CPU ■ Maybe VxLAN first, then a fully routed approach ■ https://github.com/freifunkMUC/site-ffm/issues/87
  • 20. 20 Community • Freifunk Darmstadt and Freifunk Regensburg helped a lot during development of wgkex! • B.A.T.M.A.N. developers helped a lot during debugging the performance issue and created many bugfixes • Everything is opensource and available on Github https://github.com/freifunkMUC • More background and all fixes: https://ffmuc.net/freifunkmuc/2020/12/03/wireguard-firmware/
  • 21. 21 Thanks to everyone involved • Freifunk Darmstadt @hexa • Freifunk Regensburg @MoepMan • Freifunk Hannover @aiyion, @Codefetch • systemd Yu Watanabe, Lennart Poettering • WireGuard Jason A. Donenfeld • B.A.T.M.A.N. @ecsv @T_X • All the folks of FFMUC for testing • Everyone else who we forgot and was involved in any way => Community rocks! #Together #OpenSource