SlideShare a Scribd company logo
Tech talk at UKNOF 2020 1
ffmuc presents #FFMEET
Technical challenges as well as social challenges during COVID-19
when running an open source conference system
Tech talk at UKNOF 2020 2
awlnx
● Annika Wickert
● Senior Network Engineer @AS51324
● Twitter @awlnx
krombel
● Matthias Kesler
● IT Consultant
● Twitter @kr0mbel
Who are we?
Tech talk at UKNOF 2020 3
Why did we set up jitsi?
• An upcoming Freifunk Meeting was about to get canceled because of
Corona
• The architecture of jitsi looked straight forward and easy to scale
• In the past we used NextCloud talk but we had issues (max 6. users per
conference)
− We needed something different for at least 20 - 30 people
− jitsi seemed easy and straightforward to install
• “apt install jitsi-meet” => Done ¯_(ツ)_/¯
Tech talk at UKNOF 2020 4
First test - looks promising
• Testing with some people of Freifunk Munich
• Worked like a charm, with a handful of people
• The idea was born:
o Maybe other people have the same problem?
o Why not open it for the public?
o Maybe teachers, healthcare workers, etc. need that, too?
Tech talk at UKNOF 2020 5
We need insights! Let’s monitor!
• How many users are on the platform?
• What’s the impact on the Hardware/VMs?
• Do we have bottlenecks? If so: Where?
Tech talk at UKNOF 2020 6
The project gains attention
• Interviews
• BR, DigitalCourage, Focus Online, Stimme, SZ, …
• appearing on several public server lists
Tech talk at UKNOF 2020 7
Holidays are over … school starts - 20.04.2020
• All time record before was around 600 Users
Tech talk at UKNOF 2020 8
Some unexpected things happen
People started depending on our system
• Schools & universities (not only DE)
• Small companies
• Open-Source-Communities
(repeatedly) meet on our platform
• Typo3, BitCoin, dotfmp
• In Peru and they use #FFMEET as
their primary conf system
Tech talk at UKNOF 2020 9
People tend to be in firewalled networks => TURN
• Especially after school started again we got many people complaining about
video blackouts etc.
• Reason: Firewalls/NAT/Proxy so no UDP nor direct connection possible
• sysctl -w net.ipv4.tcp_congestion_control=bbr
• Solution: TURN (Traversal Using Relays around NAT) RFC 5766 and 6156
• Acts as man in the middle to forward packets back and forth
Tech talk at UKNOF 2020 10
Of course: It did not work as we intended it to do...
• Ugly “Feature” in browsers
=> They prefer the first TURN
server in the list …
=> Patch Jitsi to randomize the
sequence
Tech talk at UKNOF 2020 11
21.04.2020 - Another record! Sadly also another bug
• 20 min. to find the issue, isolate and fix it!
• Unfortunately there was an other bug hiding after the crash
• We run at reduced capacity for a few hours but as we have so many servers
that was not an issue.
• Crash of prosody crashed some important DBs :/ ⇒ Loadbalancing failed
Tech talk at UKNOF 2020 12
We run on VMs with high PPS and many interrupts
• Oversubscribed hosts are an issue for us => pure hardware performs best
• We do many PPS and many interrupts on a single videobridge
• Tuning buffers and memory of the kernel
Tech talk at UKNOF 2020 13
Scaling the infrastructure
Tech talk at UKNOF 2020 14
People ask for bigger meetings
• Octo solves the problem of only one videobridge per conference (server-side
is no longer an issue)
• Sadly video traffic is unencrypted between videobridges => we need a
solution for that
• As we do high packet rates we need a full-mesh VPN without a central point
where all traffic has to pass
=>
Tech talk at UKNOF 2020 15
15
Octo on all bridges
Tech talk at UKNOF 2020 16
How it evolved
Tech talk at UKNOF 2020 17
Some figures? Since we started #FFMEET
• ~ 1.109 TB videotraffic
• ~ 1.407.653.191 HTTPS Requests
• ~ 360 CPU Cores / 720GB RAM
• ~ 1.100.000 Unique Users
• ~ 700 tickets solved (schools, charity projects, individuals etc.)
• 2 Admins and at least 15 people in the background for translations,
press, wiki etc.
Tech talk at UKNOF 2020 18
A special thanks to everyone who helped
writing/translating the FAQs.
Also thanks to all the users and of course our
supporters:
Thank you very much!
Tech talk at UKNOF 2020 19
Backup Slides
Tech talk at UKNOF 2020 20
We are running on the internet aren’t we?
• From time to time there are issues in the internet beyond our control (route
leaks, connectivity issues between providers, maintenance)
• We have many servers and providers so we can easily mitigate connectivity
issues
Tech talk at UKNOF 2020 21
We got more popular so DDOS attacks happen
• People attacked our infrastructure and didn’t stop when they couldn’t bring
down #FFMEET
• They started to attack other parts of our infrastructure
Tech talk at UKNOF 2020 22
Problems start - Prosody
• Change network_backend to “epoll” - No more 1024 connections as a limit
Tech talk at UKNOF 2020 23
Don’t tune only one part of the stack - nginx
• 502 all over the place
• Raise the number of nginx workers
• Raise max open files limit
Tech talk at UKNOF 2020 24
How do we update without user impact?
• We can upgrade the videobridges one by one
• We query the videobridge API and wait for 0 users then run our custom
upgrade script
• Upgrades of videobridges just happen in the background during the day/night
• Jicofo/Prosody upgrades … sadly we have to do update them manually but
we don’t need to update them very often

More Related Content

What's hot

Welcome talk unleashing the future of open-source enterprise cloud computing
Welcome talk   unleashing the future of open-source enterprise cloud computingWelcome talk   unleashing the future of open-source enterprise cloud computing
Welcome talk unleashing the future of open-source enterprise cloud computing
NETWAYS
 
Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014
Netgate
 
Talk@JanusCon2019: Janus, WebRTC and ML - Fantastic technologies and how to m...
Talk@JanusCon2019: Janus, WebRTC and ML - Fantastic technologies and how to m...Talk@JanusCon2019: Janus, WebRTC and ML - Fantastic technologies and how to m...
Talk@JanusCon2019: Janus, WebRTC and ML - Fantastic technologies and how to m...
Paolo Saviano
 
Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019
Lorenzo Miniero
 
An SFU/MCU integration for heterogeneous environments
An SFU/MCU integration for heterogeneous environmentsAn SFU/MCU integration for heterogeneous environments
An SFU/MCU integration for heterogeneous environments
Giacomo Vacca
 
Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014
Netgate
 
Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014
Netgate
 
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - KingOpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebula Project
 
WebRTC Standards & Implementation Q&A - The Internals of WebRTC Browsers Impl...
WebRTC Standards & Implementation Q&A - The Internals of WebRTC Browsers Impl...WebRTC Standards & Implementation Q&A - The Internals of WebRTC Browsers Impl...
WebRTC Standards & Implementation Q&A - The Internals of WebRTC Browsers Impl...
Amir Zmora
 
Fuzzing Janus @ IPTComm 2019
Fuzzing Janus @ IPTComm 2019Fuzzing Janus @ IPTComm 2019
Fuzzing Janus @ IPTComm 2019
Lorenzo Miniero
 
Protocol Labs, David Dias, TADSummit 2018
Protocol Labs, David Dias, TADSummit 2018Protocol Labs, David Dias, TADSummit 2018
Protocol Labs, David Dias, TADSummit 2018
Alan Quayle
 
Kamailio presence + json
Kamailio presence + jsonKamailio presence + json
Kamailio presence + json
Emmanuel Schmidbauer
 
NATS & IoT
NATS & IoTNATS & IoT
NATS & IoT
Raül Pérez
 
OAuth and STUN, TURN in WebRTC context RFC7635
OAuth and STUN, TURN  in WebRTC context RFC7635OAuth and STUN, TURN  in WebRTC context RFC7635
OAuth and STUN, TURN in WebRTC context RFC7635
Mihály Mészáros
 
Scaling WebRTC deployments with multicast @ IETF 110 MBONED
Scaling WebRTC deployments with multicast @ IETF 110 MBONEDScaling WebRTC deployments with multicast @ IETF 110 MBONED
Scaling WebRTC deployments with multicast @ IETF 110 MBONED
Lorenzo Miniero
 
Implementation Lessons using WebRTC in Asterisk
Implementation Lessons using WebRTC in AsteriskImplementation Lessons using WebRTC in Asterisk
Implementation Lessons using WebRTC in Asterisk
Moises Silva
 
Shiny New HTTP Shit
Shiny New HTTP ShitShiny New HTTP Shit
Shiny New HTTP Shit
Mark Nottingham
 
Turning live events to virtual with Janus
Turning live events to virtual with JanusTurning live events to virtual with Janus
Turning live events to virtual with Janus
Lorenzo Miniero
 
Duda I/O - Lightning Talk Google Summer of Code Summit 2014
Duda I/O - Lightning Talk Google Summer of Code Summit 2014Duda I/O - Lightning Talk Google Summer of Code Summit 2014
Duda I/O - Lightning Talk Google Summer of Code Summit 2014
Eduardo Silva Pereira
 
Virtual IETF meetings with WebRTC @ IETF 109 MOPS
Virtual IETF meetings with WebRTC @ IETF 109 MOPSVirtual IETF meetings with WebRTC @ IETF 109 MOPS
Virtual IETF meetings with WebRTC @ IETF 109 MOPS
Lorenzo Miniero
 

What's hot (20)

Welcome talk unleashing the future of open-source enterprise cloud computing
Welcome talk   unleashing the future of open-source enterprise cloud computingWelcome talk   unleashing the future of open-source enterprise cloud computing
Welcome talk unleashing the future of open-source enterprise cloud computing
 
Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014
 
Talk@JanusCon2019: Janus, WebRTC and ML - Fantastic technologies and how to m...
Talk@JanusCon2019: Janus, WebRTC and ML - Fantastic technologies and how to m...Talk@JanusCon2019: Janus, WebRTC and ML - Fantastic technologies and how to m...
Talk@JanusCon2019: Janus, WebRTC and ML - Fantastic technologies and how to m...
 
Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019
 
An SFU/MCU integration for heterogeneous environments
An SFU/MCU integration for heterogeneous environmentsAn SFU/MCU integration for heterogeneous environments
An SFU/MCU integration for heterogeneous environments
 
Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014
 
Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014
 
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - KingOpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
OpenNebulaConf2018 - Private Cloud at King - Jonathan Grahl - King
 
WebRTC Standards & Implementation Q&A - The Internals of WebRTC Browsers Impl...
WebRTC Standards & Implementation Q&A - The Internals of WebRTC Browsers Impl...WebRTC Standards & Implementation Q&A - The Internals of WebRTC Browsers Impl...
WebRTC Standards & Implementation Q&A - The Internals of WebRTC Browsers Impl...
 
Fuzzing Janus @ IPTComm 2019
Fuzzing Janus @ IPTComm 2019Fuzzing Janus @ IPTComm 2019
Fuzzing Janus @ IPTComm 2019
 
Protocol Labs, David Dias, TADSummit 2018
Protocol Labs, David Dias, TADSummit 2018Protocol Labs, David Dias, TADSummit 2018
Protocol Labs, David Dias, TADSummit 2018
 
Kamailio presence + json
Kamailio presence + jsonKamailio presence + json
Kamailio presence + json
 
NATS & IoT
NATS & IoTNATS & IoT
NATS & IoT
 
OAuth and STUN, TURN in WebRTC context RFC7635
OAuth and STUN, TURN  in WebRTC context RFC7635OAuth and STUN, TURN  in WebRTC context RFC7635
OAuth and STUN, TURN in WebRTC context RFC7635
 
Scaling WebRTC deployments with multicast @ IETF 110 MBONED
Scaling WebRTC deployments with multicast @ IETF 110 MBONEDScaling WebRTC deployments with multicast @ IETF 110 MBONED
Scaling WebRTC deployments with multicast @ IETF 110 MBONED
 
Implementation Lessons using WebRTC in Asterisk
Implementation Lessons using WebRTC in AsteriskImplementation Lessons using WebRTC in Asterisk
Implementation Lessons using WebRTC in Asterisk
 
Shiny New HTTP Shit
Shiny New HTTP ShitShiny New HTTP Shit
Shiny New HTTP Shit
 
Turning live events to virtual with Janus
Turning live events to virtual with JanusTurning live events to virtual with Janus
Turning live events to virtual with Janus
 
Duda I/O - Lightning Talk Google Summer of Code Summit 2014
Duda I/O - Lightning Talk Google Summer of Code Summit 2014Duda I/O - Lightning Talk Google Summer of Code Summit 2014
Duda I/O - Lightning Talk Google Summer of Code Summit 2014
 
Virtual IETF meetings with WebRTC @ IETF 109 MOPS
Virtual IETF meetings with WebRTC @ IETF 109 MOPSVirtual IETF meetings with WebRTC @ IETF 109 MOPS
Virtual IETF meetings with WebRTC @ IETF 109 MOPS
 

Similar to FFMUC presents #ffmeet - #virtualUKNOF

Using IT Equipment in Live Broadcast
Using IT Equipment in Live BroadcastUsing IT Equipment in Live Broadcast
Using IT Equipment in Live Broadcast
Kieran Kunhya
 
The I in Internet of Things: Implications for the Global Open Internet
The I in Internet of Things: Implications for the Global Open InternetThe I in Internet of Things: Implications for the Global Open Internet
The I in Internet of Things: Implications for the Global Open Internet
Internet Technology Matters (Internet Society)
 
DevCon 5 (July 2013) - WebSockets
DevCon 5 (July 2013) - WebSocketsDevCon 5 (July 2013) - WebSockets
DevCon 5 (July 2013) - WebSockets
Crocodile WebRTC SDK and Cloud Signalling Network
 
The Considerations for Internet of Things @ 2017
The Considerations for Internet of Things @ 2017The Considerations for Internet of Things @ 2017
The Considerations for Internet of Things @ 2017
Jian-Hong Pan
 
web-rtc presentation on TNC 2013, Technical Advisory Committee meeting
web-rtc presentation on TNC 2013, Technical Advisory Committee meetingweb-rtc presentation on TNC 2013, Technical Advisory Committee meeting
web-rtc presentation on TNC 2013, Technical Advisory Committee meeting
Jan Meijer
 
Moving to software-based production workflows and containerisation of media a...
Moving to software-based production workflows and containerisation of media a...Moving to software-based production workflows and containerisation of media a...
Moving to software-based production workflows and containerisation of media a...
Kieran Kunhya
 
Modern VoIP in Modern Infrastructures
Modern VoIP in Modern InfrastructuresModern VoIP in Modern Infrastructures
Modern VoIP in Modern Infrastructures
Giacomo Vacca
 
QoS for Media Networks
QoS for Media NetworksQoS for Media Networks
QoS for Media Networks
Amine Choukir
 
WebRTC Workshop 2013 given at the IMS World Forum
WebRTC Workshop 2013 given at the IMS World ForumWebRTC Workshop 2013 given at the IMS World Forum
WebRTC Workshop 2013 given at the IMS World Forum
Alan Quayle
 
DevCon 5 (December 2013) - WebRTC & WebSockets
DevCon 5 (December 2013) - WebRTC & WebSocketsDevCon 5 (December 2013) - WebRTC & WebSockets
DevCon 5 (December 2013) - WebRTC & WebSockets
Crocodile WebRTC SDK and Cloud Signalling Network
 
WebRTC standards update - November 2014
WebRTC standards update - November 2014WebRTC standards update - November 2014
WebRTC standards update - November 2014
Victor Pascual Ávila
 
Upperside Webinar - WebRTC Standards Update
Upperside Webinar - WebRTC Standards UpdateUpperside Webinar - WebRTC Standards Update
Upperside Webinar - WebRTC Standards Update
UppersideConferences
 
IoT: An introduction
IoT: An introductionIoT: An introduction
IoT: An introduction
JWORKS powered by Ordina
 
The fight for surviving in the IoT world
The fight for surviving in the IoT worldThe fight for surviving in the IoT world
The fight for surviving in the IoT world
Radu Vunvulea
 
The fight for surviving in the IoT world - Radu Vunvulea
The fight for surviving in the IoT world - Radu VunvuleaThe fight for surviving in the IoT world - Radu Vunvulea
The fight for surviving in the IoT world - Radu Vunvulea
ITCamp
 
WebRTC Summit (June 2014) - WebRTC Interoperability (and why it is important)
WebRTC Summit (June 2014) - WebRTC Interoperability (and why it is important)WebRTC Summit (June 2014) - WebRTC Interoperability (and why it is important)
WebRTC Summit (June 2014) - WebRTC Interoperability (and why it is important)
Crocodile WebRTC SDK and Cloud Signalling Network
 
Upperside Webinar- WebRTC from the service provider prism-final
Upperside Webinar- WebRTC from the service provider prism-finalUpperside Webinar- WebRTC from the service provider prism-final
Upperside Webinar- WebRTC from the service provider prism-final
Amir Zmora
 
An hour with WebRTC FIC UDC
An hour with WebRTC FIC UDCAn hour with WebRTC FIC UDC
An hour with WebRTC FIC UDC
Quobis
 
Tf web rtc-berlin-mai2016-sa8t2 roadmap-janmeijer
Tf web rtc-berlin-mai2016-sa8t2 roadmap-janmeijerTf web rtc-berlin-mai2016-sa8t2 roadmap-janmeijer
Tf web rtc-berlin-mai2016-sa8t2 roadmap-janmeijer
Jan Meijer
 
SFScon 21 - Roberto Innocenti - PPC64 Open Hardware Notebook prototype around...
SFScon 21 - Roberto Innocenti - PPC64 Open Hardware Notebook prototype around...SFScon 21 - Roberto Innocenti - PPC64 Open Hardware Notebook prototype around...
SFScon 21 - Roberto Innocenti - PPC64 Open Hardware Notebook prototype around...
South Tyrol Free Software Conference
 

Similar to FFMUC presents #ffmeet - #virtualUKNOF (20)

Using IT Equipment in Live Broadcast
Using IT Equipment in Live BroadcastUsing IT Equipment in Live Broadcast
Using IT Equipment in Live Broadcast
 
The I in Internet of Things: Implications for the Global Open Internet
The I in Internet of Things: Implications for the Global Open InternetThe I in Internet of Things: Implications for the Global Open Internet
The I in Internet of Things: Implications for the Global Open Internet
 
DevCon 5 (July 2013) - WebSockets
DevCon 5 (July 2013) - WebSocketsDevCon 5 (July 2013) - WebSockets
DevCon 5 (July 2013) - WebSockets
 
The Considerations for Internet of Things @ 2017
The Considerations for Internet of Things @ 2017The Considerations for Internet of Things @ 2017
The Considerations for Internet of Things @ 2017
 
web-rtc presentation on TNC 2013, Technical Advisory Committee meeting
web-rtc presentation on TNC 2013, Technical Advisory Committee meetingweb-rtc presentation on TNC 2013, Technical Advisory Committee meeting
web-rtc presentation on TNC 2013, Technical Advisory Committee meeting
 
Moving to software-based production workflows and containerisation of media a...
Moving to software-based production workflows and containerisation of media a...Moving to software-based production workflows and containerisation of media a...
Moving to software-based production workflows and containerisation of media a...
 
Modern VoIP in Modern Infrastructures
Modern VoIP in Modern InfrastructuresModern VoIP in Modern Infrastructures
Modern VoIP in Modern Infrastructures
 
QoS for Media Networks
QoS for Media NetworksQoS for Media Networks
QoS for Media Networks
 
WebRTC Workshop 2013 given at the IMS World Forum
WebRTC Workshop 2013 given at the IMS World ForumWebRTC Workshop 2013 given at the IMS World Forum
WebRTC Workshop 2013 given at the IMS World Forum
 
DevCon 5 (December 2013) - WebRTC & WebSockets
DevCon 5 (December 2013) - WebRTC & WebSocketsDevCon 5 (December 2013) - WebRTC & WebSockets
DevCon 5 (December 2013) - WebRTC & WebSockets
 
WebRTC standards update - November 2014
WebRTC standards update - November 2014WebRTC standards update - November 2014
WebRTC standards update - November 2014
 
Upperside Webinar - WebRTC Standards Update
Upperside Webinar - WebRTC Standards UpdateUpperside Webinar - WebRTC Standards Update
Upperside Webinar - WebRTC Standards Update
 
IoT: An introduction
IoT: An introductionIoT: An introduction
IoT: An introduction
 
The fight for surviving in the IoT world
The fight for surviving in the IoT worldThe fight for surviving in the IoT world
The fight for surviving in the IoT world
 
The fight for surviving in the IoT world - Radu Vunvulea
The fight for surviving in the IoT world - Radu VunvuleaThe fight for surviving in the IoT world - Radu Vunvulea
The fight for surviving in the IoT world - Radu Vunvulea
 
WebRTC Summit (June 2014) - WebRTC Interoperability (and why it is important)
WebRTC Summit (June 2014) - WebRTC Interoperability (and why it is important)WebRTC Summit (June 2014) - WebRTC Interoperability (and why it is important)
WebRTC Summit (June 2014) - WebRTC Interoperability (and why it is important)
 
Upperside Webinar- WebRTC from the service provider prism-final
Upperside Webinar- WebRTC from the service provider prism-finalUpperside Webinar- WebRTC from the service provider prism-final
Upperside Webinar- WebRTC from the service provider prism-final
 
An hour with WebRTC FIC UDC
An hour with WebRTC FIC UDCAn hour with WebRTC FIC UDC
An hour with WebRTC FIC UDC
 
Tf web rtc-berlin-mai2016-sa8t2 roadmap-janmeijer
Tf web rtc-berlin-mai2016-sa8t2 roadmap-janmeijerTf web rtc-berlin-mai2016-sa8t2 roadmap-janmeijer
Tf web rtc-berlin-mai2016-sa8t2 roadmap-janmeijer
 
SFScon 21 - Roberto Innocenti - PPC64 Open Hardware Notebook prototype around...
SFScon 21 - Roberto Innocenti - PPC64 Open Hardware Notebook prototype around...SFScon 21 - Roberto Innocenti - PPC64 Open Hardware Notebook prototype around...
SFScon 21 - Roberto Innocenti - PPC64 Open Hardware Notebook prototype around...
 

Recently uploaded

MARY JANE WILSON, A “BOA MÃE” .
MARY JANE WILSON, A “BOA MÃE”           .MARY JANE WILSON, A “BOA MÃE”           .
MARY JANE WILSON, A “BOA MÃE” .
Colégio Santa Teresinha
 
Hindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdfHindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdf
Dr. Mulla Adam Ali
 
Main Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docxMain Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docx
adhitya5119
 
writing about opinions about Australia the movie
writing about opinions about Australia the moviewriting about opinions about Australia the movie
writing about opinions about Australia the movie
Nicholas Montgomery
 
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) CurriculumPhilippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
MJDuyan
 
A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
Jean Carlos Nunes Paixão
 
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdfWalmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
TechSoup
 
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching AptitudeUGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
S. Raj Kumar
 
ZK on Polkadot zero knowledge proofs - sub0.pptx
ZK on Polkadot zero knowledge proofs - sub0.pptxZK on Polkadot zero knowledge proofs - sub0.pptx
ZK on Polkadot zero knowledge proofs - sub0.pptx
dot55audits
 
BBR 2024 Summer Sessions Interview Training
BBR  2024 Summer Sessions Interview TrainingBBR  2024 Summer Sessions Interview Training
BBR 2024 Summer Sessions Interview Training
Katrina Pritchard
 
Bed Making ( Introduction, Purpose, Types, Articles, Scientific principles, N...
Bed Making ( Introduction, Purpose, Types, Articles, Scientific principles, N...Bed Making ( Introduction, Purpose, Types, Articles, Scientific principles, N...
Bed Making ( Introduction, Purpose, Types, Articles, Scientific principles, N...
Leena Ghag-Sakpal
 
Solutons Maths Escape Room Spatial .pptx
Solutons Maths Escape Room Spatial .pptxSolutons Maths Escape Room Spatial .pptx
Solutons Maths Escape Room Spatial .pptx
spdendr
 
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
Nguyen Thanh Tu Collection
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
Nguyen Thanh Tu Collection
 
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptxC1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
mulvey2
 
How to Create a More Engaging and Human Online Learning Experience
How to Create a More Engaging and Human Online Learning Experience How to Create a More Engaging and Human Online Learning Experience
How to Create a More Engaging and Human Online Learning Experience
Wahiba Chair Training & Consulting
 
clinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdfclinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdf
Priyankaranawat4
 
math operations ued in python and all used
math operations ued in python and all usedmath operations ued in python and all used
math operations ued in python and all used
ssuser13ffe4
 
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptxNEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
iammrhaywood
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
Nicholas Montgomery
 

Recently uploaded (20)

MARY JANE WILSON, A “BOA MÃE” .
MARY JANE WILSON, A “BOA MÃE”           .MARY JANE WILSON, A “BOA MÃE”           .
MARY JANE WILSON, A “BOA MÃE” .
 
Hindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdfHindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdf
 
Main Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docxMain Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docx
 
writing about opinions about Australia the movie
writing about opinions about Australia the moviewriting about opinions about Australia the movie
writing about opinions about Australia the movie
 
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) CurriculumPhilippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
 
A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
 
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdfWalmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
 
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching AptitudeUGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
 
ZK on Polkadot zero knowledge proofs - sub0.pptx
ZK on Polkadot zero knowledge proofs - sub0.pptxZK on Polkadot zero knowledge proofs - sub0.pptx
ZK on Polkadot zero knowledge proofs - sub0.pptx
 
BBR 2024 Summer Sessions Interview Training
BBR  2024 Summer Sessions Interview TrainingBBR  2024 Summer Sessions Interview Training
BBR 2024 Summer Sessions Interview Training
 
Bed Making ( Introduction, Purpose, Types, Articles, Scientific principles, N...
Bed Making ( Introduction, Purpose, Types, Articles, Scientific principles, N...Bed Making ( Introduction, Purpose, Types, Articles, Scientific principles, N...
Bed Making ( Introduction, Purpose, Types, Articles, Scientific principles, N...
 
Solutons Maths Escape Room Spatial .pptx
Solutons Maths Escape Room Spatial .pptxSolutons Maths Escape Room Spatial .pptx
Solutons Maths Escape Room Spatial .pptx
 
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
 
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptxC1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
 
How to Create a More Engaging and Human Online Learning Experience
How to Create a More Engaging and Human Online Learning Experience How to Create a More Engaging and Human Online Learning Experience
How to Create a More Engaging and Human Online Learning Experience
 
clinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdfclinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdf
 
math operations ued in python and all used
math operations ued in python and all usedmath operations ued in python and all used
math operations ued in python and all used
 
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptxNEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
 

FFMUC presents #ffmeet - #virtualUKNOF

  • 1. Tech talk at UKNOF 2020 1 ffmuc presents #FFMEET Technical challenges as well as social challenges during COVID-19 when running an open source conference system
  • 2. Tech talk at UKNOF 2020 2 awlnx ● Annika Wickert ● Senior Network Engineer @AS51324 ● Twitter @awlnx krombel ● Matthias Kesler ● IT Consultant ● Twitter @kr0mbel Who are we?
  • 3. Tech talk at UKNOF 2020 3 Why did we set up jitsi? • An upcoming Freifunk Meeting was about to get canceled because of Corona • The architecture of jitsi looked straight forward and easy to scale • In the past we used NextCloud talk but we had issues (max 6. users per conference) − We needed something different for at least 20 - 30 people − jitsi seemed easy and straightforward to install • “apt install jitsi-meet” => Done ¯_(ツ)_/¯
  • 4. Tech talk at UKNOF 2020 4 First test - looks promising • Testing with some people of Freifunk Munich • Worked like a charm, with a handful of people • The idea was born: o Maybe other people have the same problem? o Why not open it for the public? o Maybe teachers, healthcare workers, etc. need that, too?
  • 5. Tech talk at UKNOF 2020 5 We need insights! Let’s monitor! • How many users are on the platform? • What’s the impact on the Hardware/VMs? • Do we have bottlenecks? If so: Where?
  • 6. Tech talk at UKNOF 2020 6 The project gains attention • Interviews • BR, DigitalCourage, Focus Online, Stimme, SZ, … • appearing on several public server lists
  • 7. Tech talk at UKNOF 2020 7 Holidays are over … school starts - 20.04.2020 • All time record before was around 600 Users
  • 8. Tech talk at UKNOF 2020 8 Some unexpected things happen People started depending on our system • Schools & universities (not only DE) • Small companies • Open-Source-Communities (repeatedly) meet on our platform • Typo3, BitCoin, dotfmp • In Peru and they use #FFMEET as their primary conf system
  • 9. Tech talk at UKNOF 2020 9 People tend to be in firewalled networks => TURN • Especially after school started again we got many people complaining about video blackouts etc. • Reason: Firewalls/NAT/Proxy so no UDP nor direct connection possible • sysctl -w net.ipv4.tcp_congestion_control=bbr • Solution: TURN (Traversal Using Relays around NAT) RFC 5766 and 6156 • Acts as man in the middle to forward packets back and forth
  • 10. Tech talk at UKNOF 2020 10 Of course: It did not work as we intended it to do... • Ugly “Feature” in browsers => They prefer the first TURN server in the list … => Patch Jitsi to randomize the sequence
  • 11. Tech talk at UKNOF 2020 11 21.04.2020 - Another record! Sadly also another bug • 20 min. to find the issue, isolate and fix it! • Unfortunately there was an other bug hiding after the crash • We run at reduced capacity for a few hours but as we have so many servers that was not an issue. • Crash of prosody crashed some important DBs :/ ⇒ Loadbalancing failed
  • 12. Tech talk at UKNOF 2020 12 We run on VMs with high PPS and many interrupts • Oversubscribed hosts are an issue for us => pure hardware performs best • We do many PPS and many interrupts on a single videobridge • Tuning buffers and memory of the kernel
  • 13. Tech talk at UKNOF 2020 13 Scaling the infrastructure
  • 14. Tech talk at UKNOF 2020 14 People ask for bigger meetings • Octo solves the problem of only one videobridge per conference (server-side is no longer an issue) • Sadly video traffic is unencrypted between videobridges => we need a solution for that • As we do high packet rates we need a full-mesh VPN without a central point where all traffic has to pass =>
  • 15. Tech talk at UKNOF 2020 15 15 Octo on all bridges
  • 16. Tech talk at UKNOF 2020 16 How it evolved
  • 17. Tech talk at UKNOF 2020 17 Some figures? Since we started #FFMEET • ~ 1.109 TB videotraffic • ~ 1.407.653.191 HTTPS Requests • ~ 360 CPU Cores / 720GB RAM • ~ 1.100.000 Unique Users • ~ 700 tickets solved (schools, charity projects, individuals etc.) • 2 Admins and at least 15 people in the background for translations, press, wiki etc.
  • 18. Tech talk at UKNOF 2020 18 A special thanks to everyone who helped writing/translating the FAQs. Also thanks to all the users and of course our supporters: Thank you very much!
  • 19. Tech talk at UKNOF 2020 19 Backup Slides
  • 20. Tech talk at UKNOF 2020 20 We are running on the internet aren’t we? • From time to time there are issues in the internet beyond our control (route leaks, connectivity issues between providers, maintenance) • We have many servers and providers so we can easily mitigate connectivity issues
  • 21. Tech talk at UKNOF 2020 21 We got more popular so DDOS attacks happen • People attacked our infrastructure and didn’t stop when they couldn’t bring down #FFMEET • They started to attack other parts of our infrastructure
  • 22. Tech talk at UKNOF 2020 22 Problems start - Prosody • Change network_backend to “epoll” - No more 1024 connections as a limit
  • 23. Tech talk at UKNOF 2020 23 Don’t tune only one part of the stack - nginx • 502 all over the place • Raise the number of nginx workers • Raise max open files limit
  • 24. Tech talk at UKNOF 2020 24 How do we update without user impact? • We can upgrade the videobridges one by one • We query the videobridge API and wait for 0 users then run our custom upgrade script • Upgrades of videobridges just happen in the background during the day/night • Jicofo/Prosody upgrades … sadly we have to do update them manually but we don’t need to update them very often