SlideShare a Scribd company logo
ffmuc: FFMEET running a non-profit
conference system
3 months later
awlnx
● Annika Wickert
● Senior Network Engineer @AS51324
● Twitter @awlnx
Krombel
● Matthias Kesler
● IT Consultant
● Twitter @kr0mbel
2
Who are we?
3
Jitsi to close the social gap during corona
• An upcoming Freifunk Meeting was about to get canceled because of Corona
• In the past we used NextCloud talk but we had issues (max 6. users per
conference)
■ We needed something different for at least 20 - 30 people
■ jitsi seemed easy and straightforward to install
● “apt install jitsi-meet” => Done ¯_(ツ)_/¯
4
Debian package wins over Docker
• We had some issues with docker-proxy in the past
• docker image didn’t look up to date
• With scaling in mind
■ Planned using dedicated hardware for Jitsi anyway
■ Debian packages are available
5
First test - looks promising
• Testing with some people of Freifunk Munich
• Worked like a charm, with a handful of people
• The idea was born:
■ Maybe other people have the same problem?
■ Why not open it for the public?
■ Maybe teachers, healthcare workers, etc. need that, too?
6
We need insights! Let’s monitor!
• How many users are on the platform?
• What’s the impact on the Hardware/VMs?
• Do we have bottlenecks? If so: Where?
7
What we needed for success
• First servers got donated by individuals (max 4 cores)
■ Went well as long as we only had small conferences
• Schools started to use our infrastructure => cheap servers were not
enough anymore
→ Videobridges need powerful hardware and we may need many of them!
→ Looking for sponsors
8
Scaling the infrastructure
9
The project gains attention
• Interviews
• BR, DigitalCourage, Focus Online, Stimme, SZ, …
• diverse Listen
10
Problems start - Prosody
• Change network_backend to “epoll” - No more 1024 connections as a limit
11
Don’t tune only one part of the stack - nginx
• 502 all over the place
• Raise the number of nginx workers
• Raise max open files limit
12
How do we update without user impact?
• We can upgrade the videobridges one by one
• We query the videobridge API and wait for 0 users then run our custom
upgrade script
• Upgrades of videobridges just happen in the background during the day/night
• Jicofo/Prosody upgrades … sadly we have to do them manually but we don’t
need to update them very often
13
Holidays are over … school starts - 20.04.2020
• All time record before was around 600 Users
14
21.04.2020 - Another record! Sadly also another bug
● 20 min. to find the issue, isolate and fix it!
● Unfortunately there was an other bug hiding after the crash
● We run at reduced capacity for a few hours but as we have so many servers
that was not an issue.
● Crash of prosody crashed some important DBs :/ ⇒ Loadbalancing failed
15
New records every day!
• On 29.04.2020 we hit another highscore of 1503 concurrent users. Servers had
no issues!
16
People tend to be in firewalled networks => TURN
• To connect devices which cannot properly transmit data to the videobridge
(UDP blocked)
• videobridge2 used to provide a TCP Fallback to 443 … but it’s not real
HTTPS traffic so NGFW tend to block it
• Solution TURN (Traversal Using Relays around NAT) RFC5766 and 6156
• Ugly “Feature” in browsers => They prefer the first turnserver in the list … =>
Patch Jitsi to randomize the sequence
17
People ask for bigger meetings
• Octo solves the problem of only one videobridge per conference (server-side
is no longer an issue)
• Sadly video traffic is unencrypted between videobridges => we need a
solution for that
• As we do high packet rates we need a full-mesh VPN without a central point
where all traffic has to pass
=>
18
82 people in one meeting and nebula in action
19
Octo on all bridges
20
Bugs with Octo
• Videobridges join XMPP via pubsub and MUC => no region info in pubsub
• Update to jitsi-videobridge2 unstable
21
We are running on the internet aren’t we?
• From time to time there are issues in the internet beyond our control (route
leaks, connectivity issues between providers, maintenance)
• We have many servers and providers so we can easily mitigate connectivity
issues
22
We got more popular so DDOS attacks happen
• People attacked our infrastructure and didn’t stop when they couldn’t bring
down #FFMEET
• They started to attack other parts of our infrastructure
23
Custom code build on upstream/master
• We no longer run the upstream packages for most software in the stack
• We run a fork from latest master with some additions
• jitsi-videobridge2 is still the latest unstable package on all hosts
• We forked the electron app
24
Our infrastructure 3 months ago!
25
How it evolved
26
Some figures? 3 Months of #FFMEET
• ~ 810TB videotraffic
• ~ 847.955.871 HTTPS Requests
• ~ 360 CPU Cores / 720GB of RAM
• ~ 800.000 Unique Users
• ~ 500 tickets solved (many schools, charity projects, individuals etc.)
• ~ 200 Euro / day
• 2 Admins and at least 15 people in the background for translations, press,
wiki etc
27
Some unexpected things happen
• You get famous in Peru and they use
#FFMEET as their primary conf system
• Schools start to depend on you
• Small companies start to depend on
you
28
What’s next?
• Hopefully more happy users!
• Establishing a long term roadmap for the service
• Gathering money and resources to secure the funding of the project even
more
• Every penny helps! paypal.me/ffmucspenden
29
Thank you very much!
A special thanks to all who helped
writing/translating the FAQs.
Also thank you to all the users and of
course our supporters:
30
Firefox breaks user experience
• Users experience problems especially in large conferences (25-40 people)
• Add warning for Firefox users … it’s mostly ignored … so problems still exists
• Deactivate Firefox support and release FreifunkMeet app
→https://github.com/freifunkMUC/jitsi-meet-electron/releases
31
DoH / DoT Public Resolver
• As DNS is an unencrypted protocol there are approaches to develop a more
secure one
• We offer DoH (native in Firefox) / DoT (native since Android 9) / DNSCrypt

More Related Content

What's hot

FreeSWITCH as a Microservice
FreeSWITCH as a MicroserviceFreeSWITCH as a Microservice
FreeSWITCH as a Microservice
Evan McGee
 
WebRTC & Asterisk 11
WebRTC & Asterisk 11WebRTC & Asterisk 11
WebRTC & Asterisk 11
Sanjay Willie
 
Netty - a pragmatic introduction
Netty - a pragmatic introductionNetty - a pragmatic introduction
Netty - a pragmatic introduction
Raphael Stary
 
Janus/HOMER/HEPIC @ OpenSIPS18
Janus/HOMER/HEPIC @ OpenSIPS18Janus/HOMER/HEPIC @ OpenSIPS18
Janus/HOMER/HEPIC @ OpenSIPS18
Lorenzo Miniero
 
CentOS and OpenNebula, a Perfect Match
CentOS and OpenNebula, a Perfect MatchCentOS and OpenNebula, a Perfect Match
CentOS and OpenNebula, a Perfect Match
NETWAYS
 
Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014
Netgate
 
Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014
Netgate
 
Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019
Lorenzo Miniero
 
Opening of Cloud Native Taiwan User Group Meetup#2
Opening of Cloud Native Taiwan User Group Meetup#2Opening of Cloud Native Taiwan User Group Meetup#2
Opening of Cloud Native Taiwan User Group Meetup#2
HungWei Chiu
 
Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014
Netgate
 
LF_OVS_17_State of the OVN
LF_OVS_17_State of the OVNLF_OVS_17_State of the OVN
LF_OVS_17_State of the OVN
LF_OpenvSwitch
 
OAuth and STUN, TURN in WebRTC context RFC7635
OAuth and STUN, TURN  in WebRTC context RFC7635OAuth and STUN, TURN  in WebRTC context RFC7635
OAuth and STUN, TURN in WebRTC context RFC7635
Mihály Mészáros
 
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
ShapeBlue
 
How Can OpenNebula Fit Your Needs: A European Project Feedback
How Can OpenNebula Fit Your Needs: A European Project FeedbackHow Can OpenNebula Fit Your Needs: A European Project Feedback
How Can OpenNebula Fit Your Needs: A European Project Feedback
NETWAYS
 
ONUG Tutorial: Bridges and Tunnels Drive Through OpenStack Networking
ONUG Tutorial: Bridges and Tunnels Drive Through OpenStack NetworkingONUG Tutorial: Bridges and Tunnels Drive Through OpenStack Networking
ONUG Tutorial: Bridges and Tunnels Drive Through OpenStack Networking
markmcclain
 
Fixing Docker networking - Milos Gajdos at #DOXLON
Fixing Docker networking - Milos Gajdos at #DOXLONFixing Docker networking - Milos Gajdos at #DOXLON
Fixing Docker networking - Milos Gajdos at #DOXLON
Outlyer
 
Simplifying the OpenStack and Kubernetes network stack with Romana
Simplifying the OpenStack and Kubernetes network stack with RomanaSimplifying the OpenStack and Kubernetes network stack with Romana
Simplifying the OpenStack and Kubernetes network stack with Romana
Juergen Brendel
 
Building a Small DC
Building a Small DCBuilding a Small DC
Building a Small DC
APNIC
 
Saltconf 2016: Salt stack transport and concurrency
Saltconf 2016: Salt stack transport and concurrencySaltconf 2016: Salt stack transport and concurrency
Saltconf 2016: Salt stack transport and concurrency
Thomas Jackson
 
Janus & docker: friends or foe
Janus & docker: friends or foe Janus & docker: friends or foe
Janus & docker: friends or foe
Alessandro Amirante
 

What's hot (20)

FreeSWITCH as a Microservice
FreeSWITCH as a MicroserviceFreeSWITCH as a Microservice
FreeSWITCH as a Microservice
 
WebRTC & Asterisk 11
WebRTC & Asterisk 11WebRTC & Asterisk 11
WebRTC & Asterisk 11
 
Netty - a pragmatic introduction
Netty - a pragmatic introductionNetty - a pragmatic introduction
Netty - a pragmatic introduction
 
Janus/HOMER/HEPIC @ OpenSIPS18
Janus/HOMER/HEPIC @ OpenSIPS18Janus/HOMER/HEPIC @ OpenSIPS18
Janus/HOMER/HEPIC @ OpenSIPS18
 
CentOS and OpenNebula, a Perfect Match
CentOS and OpenNebula, a Perfect MatchCentOS and OpenNebula, a Perfect Match
CentOS and OpenNebula, a Perfect Match
 
Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014Firewalls and Virtualization - pfSense Hangout June 2014
Firewalls and Virtualization - pfSense Hangout June 2014
 
Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014Network Address Translation - pfSense Hangout July 2014
Network Address Translation - pfSense Hangout July 2014
 
Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019Multistream in Janus @ CommCon 2019
Multistream in Janus @ CommCon 2019
 
Opening of Cloud Native Taiwan User Group Meetup#2
Opening of Cloud Native Taiwan User Group Meetup#2Opening of Cloud Native Taiwan User Group Meetup#2
Opening of Cloud Native Taiwan User Group Meetup#2
 
Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014Firewall and NAT Fundamentals - pfSense Hangout January 2014
Firewall and NAT Fundamentals - pfSense Hangout January 2014
 
LF_OVS_17_State of the OVN
LF_OVS_17_State of the OVNLF_OVS_17_State of the OVN
LF_OVS_17_State of the OVN
 
OAuth and STUN, TURN in WebRTC context RFC7635
OAuth and STUN, TURN  in WebRTC context RFC7635OAuth and STUN, TURN  in WebRTC context RFC7635
OAuth and STUN, TURN in WebRTC context RFC7635
 
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
 
How Can OpenNebula Fit Your Needs: A European Project Feedback
How Can OpenNebula Fit Your Needs: A European Project FeedbackHow Can OpenNebula Fit Your Needs: A European Project Feedback
How Can OpenNebula Fit Your Needs: A European Project Feedback
 
ONUG Tutorial: Bridges and Tunnels Drive Through OpenStack Networking
ONUG Tutorial: Bridges and Tunnels Drive Through OpenStack NetworkingONUG Tutorial: Bridges and Tunnels Drive Through OpenStack Networking
ONUG Tutorial: Bridges and Tunnels Drive Through OpenStack Networking
 
Fixing Docker networking - Milos Gajdos at #DOXLON
Fixing Docker networking - Milos Gajdos at #DOXLONFixing Docker networking - Milos Gajdos at #DOXLON
Fixing Docker networking - Milos Gajdos at #DOXLON
 
Simplifying the OpenStack and Kubernetes network stack with Romana
Simplifying the OpenStack and Kubernetes network stack with RomanaSimplifying the OpenStack and Kubernetes network stack with Romana
Simplifying the OpenStack and Kubernetes network stack with Romana
 
Building a Small DC
Building a Small DCBuilding a Small DC
Building a Small DC
 
Saltconf 2016: Salt stack transport and concurrency
Saltconf 2016: Salt stack transport and concurrencySaltconf 2016: Salt stack transport and concurrency
Saltconf 2016: Salt stack transport and concurrency
 
Janus & docker: friends or foe
Janus & docker: friends or foe Janus & docker: friends or foe
Janus & docker: friends or foe
 

Similar to FFMEET: running a non-profit conference system

Introducing Container Technology to TSUBAME3.0 Supercomputer
Introducing Container Technology to TSUBAME3.0 SupercomputerIntroducing Container Technology to TSUBAME3.0 Supercomputer
Introducing Container Technology to TSUBAME3.0 Supercomputer
Akihiro Nomura
 
The world is not black and white – Impact of decisions over the lifetime of a...
The world is not black and white – Impact of decisions over the lifetime of a...The world is not black and white – Impact of decisions over the lifetime of a...
The world is not black and white – Impact of decisions over the lifetime of a...
Eric Reiche
 
Using IT Equipment in Live Broadcast
Using IT Equipment in Live BroadcastUsing IT Equipment in Live Broadcast
Using IT Equipment in Live Broadcast
Kieran Kunhya
 
Gears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development ChallengesGears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development Challenges
Perforce
 
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
 
OSMC 2022 | Let’s build a private cloud – how hard can it be? by Kevin Honka
OSMC 2022 | Let’s build a private cloud – how hard can it be? by Kevin HonkaOSMC 2022 | Let’s build a private cloud – how hard can it be? by Kevin Honka
OSMC 2022 | Let’s build a private cloud – how hard can it be? by Kevin Honka
NETWAYS
 
ClueCon 2017
ClueCon 2017ClueCon 2017
ClueCon 2017
Luca Pradovera
 
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Larry Smarr
 
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Larry Smarr
 
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Larry Smarr
 
IPv6 Development in ITB 2013
IPv6 Development in ITB 2013IPv6 Development in ITB 2013
IPv6 Development in ITB 2013
Affan Basalamah
 
UPNext
UPNextUPNext
Tstat conext
Tstat conextTstat conext
Tstat conext
Danilo Giordano
 
Vimeo and Open Source (SMPTE Forum 2015)
Vimeo and Open Source (SMPTE Forum 2015)Vimeo and Open Source (SMPTE Forum 2015)
Vimeo and Open Source (SMPTE Forum 2015)
Derek Buitenhuis
 
Everyone wants (someone else) to do it: writing documentation for open source...
Everyone wants (someone else) to do it: writing documentation for open source...Everyone wants (someone else) to do it: writing documentation for open source...
Everyone wants (someone else) to do it: writing documentation for open source...
Jody Garnett
 
Fred - Simple, Smart and Swift ECM - webinar V2.3
Fred - Simple, Smart and Swift ECM - webinar V2.3Fred - Simple, Smart and Swift ECM - webinar V2.3
Fred - Simple, Smart and Swift ECM - webinar V2.3
XeniT Solutions nv
 
Moving to software-based production workflows and containerisation of media a...
Moving to software-based production workflows and containerisation of media a...Moving to software-based production workflows and containerisation of media a...
Moving to software-based production workflows and containerisation of media a...
Kieran Kunhya
 
PLNOG 9: Ron Broersma - Enterprise IPv6 Deployment
PLNOG 9: Ron Broersma - Enterprise IPv6 Deployment PLNOG 9: Ron Broersma - Enterprise IPv6 Deployment
PLNOG 9: Ron Broersma - Enterprise IPv6 Deployment
PROIDEA
 
Distributed systems
Distributed systemsDistributed systems
Distributed systems
Syed Zaid Irshad
 
MJC 2021: "Debugging Java Microservices Running on Kubernetes with Telepresence"
MJC 2021: "Debugging Java Microservices Running on Kubernetes with Telepresence"MJC 2021: "Debugging Java Microservices Running on Kubernetes with Telepresence"
MJC 2021: "Debugging Java Microservices Running on Kubernetes with Telepresence"
Daniel Bryant
 

Similar to FFMEET: running a non-profit conference system (20)

Introducing Container Technology to TSUBAME3.0 Supercomputer
Introducing Container Technology to TSUBAME3.0 SupercomputerIntroducing Container Technology to TSUBAME3.0 Supercomputer
Introducing Container Technology to TSUBAME3.0 Supercomputer
 
The world is not black and white – Impact of decisions over the lifetime of a...
The world is not black and white – Impact of decisions over the lifetime of a...The world is not black and white – Impact of decisions over the lifetime of a...
The world is not black and white – Impact of decisions over the lifetime of a...
 
Using IT Equipment in Live Broadcast
Using IT Equipment in Live BroadcastUsing IT Equipment in Live Broadcast
Using IT Equipment in Live Broadcast
 
Gears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development ChallengesGears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development Challenges
 
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
 
OSMC 2022 | Let’s build a private cloud – how hard can it be? by Kevin Honka
OSMC 2022 | Let’s build a private cloud – how hard can it be? by Kevin HonkaOSMC 2022 | Let’s build a private cloud – how hard can it be? by Kevin Honka
OSMC 2022 | Let’s build a private cloud – how hard can it be? by Kevin Honka
 
ClueCon 2017
ClueCon 2017ClueCon 2017
ClueCon 2017
 
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
 
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
 
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...
 
IPv6 Development in ITB 2013
IPv6 Development in ITB 2013IPv6 Development in ITB 2013
IPv6 Development in ITB 2013
 
UPNext
UPNextUPNext
UPNext
 
Tstat conext
Tstat conextTstat conext
Tstat conext
 
Vimeo and Open Source (SMPTE Forum 2015)
Vimeo and Open Source (SMPTE Forum 2015)Vimeo and Open Source (SMPTE Forum 2015)
Vimeo and Open Source (SMPTE Forum 2015)
 
Everyone wants (someone else) to do it: writing documentation for open source...
Everyone wants (someone else) to do it: writing documentation for open source...Everyone wants (someone else) to do it: writing documentation for open source...
Everyone wants (someone else) to do it: writing documentation for open source...
 
Fred - Simple, Smart and Swift ECM - webinar V2.3
Fred - Simple, Smart and Swift ECM - webinar V2.3Fred - Simple, Smart and Swift ECM - webinar V2.3
Fred - Simple, Smart and Swift ECM - webinar V2.3
 
Moving to software-based production workflows and containerisation of media a...
Moving to software-based production workflows and containerisation of media a...Moving to software-based production workflows and containerisation of media a...
Moving to software-based production workflows and containerisation of media a...
 
PLNOG 9: Ron Broersma - Enterprise IPv6 Deployment
PLNOG 9: Ron Broersma - Enterprise IPv6 Deployment PLNOG 9: Ron Broersma - Enterprise IPv6 Deployment
PLNOG 9: Ron Broersma - Enterprise IPv6 Deployment
 
Distributed systems
Distributed systemsDistributed systems
Distributed systems
 
MJC 2021: "Debugging Java Microservices Running on Kubernetes with Telepresence"
MJC 2021: "Debugging Java Microservices Running on Kubernetes with Telepresence"MJC 2021: "Debugging Java Microservices Running on Kubernetes with Telepresence"
MJC 2021: "Debugging Java Microservices Running on Kubernetes with Telepresence"
 

Recently uploaded

STUDY ON THE DEVELOPMENT STRATEGY OF HUZHOU TOURISM
STUDY ON THE DEVELOPMENT STRATEGY OF HUZHOU TOURISMSTUDY ON THE DEVELOPMENT STRATEGY OF HUZHOU TOURISM
STUDY ON THE DEVELOPMENT STRATEGY OF HUZHOU TOURISM
AJHSSR Journal
 
UR BHATTI ACADEMY AND ONLINE COURSES.pdf
UR BHATTI ACADEMY AND ONLINE COURSES.pdfUR BHATTI ACADEMY AND ONLINE COURSES.pdf
UR BHATTI ACADEMY AND ONLINE COURSES.pdf
urbhattiacademy
 
快速办理(worcester毕业证书)伍斯特大学毕业证PDF成绩单一模一样
快速办理(worcester毕业证书)伍斯特大学毕业证PDF成绩单一模一样快速办理(worcester毕业证书)伍斯特大学毕业证PDF成绩单一模一样
快速办理(worcester毕业证书)伍斯特大学毕业证PDF成绩单一模一样
9u4xjk4w
 
Dominate Reddit Discussions.............
Dominate Reddit Discussions.............Dominate Reddit Discussions.............
Dominate Reddit Discussions.............
SocioCosmos
 
一比一原版(AU毕业证)英国阿伯丁大学毕业证如何办理
一比一原版(AU毕业证)英国阿伯丁大学毕业证如何办理一比一原版(AU毕业证)英国阿伯丁大学毕业证如何办理
一比一原版(AU毕业证)英国阿伯丁大学毕业证如何办理
anubug
 
原版制作(Hull毕业证书)赫尔大学毕业证Offer一模一样
原版制作(Hull毕业证书)赫尔大学毕业证Offer一模一样原版制作(Hull毕业证书)赫尔大学毕业证Offer一模一样
原版制作(Hull毕业证书)赫尔大学毕业证Offer一模一样
7lkkjxt
 
HMS Facebook Stories All V1 06092024.docx
HMS Facebook Stories All V1 06092024.docxHMS Facebook Stories All V1 06092024.docx
HMS Facebook Stories All V1 06092024.docx
Charles Bayless
 
The Impact of Work Stress and Digital Literacy on Employee Performance at PT ...
The Impact of Work Stress and Digital Literacy on Employee Performance at PT ...The Impact of Work Stress and Digital Literacy on Employee Performance at PT ...
The Impact of Work Stress and Digital Literacy on Employee Performance at PT ...
AJHSSR Journal
 
Factors affecting undergraduate students’ motivation at a university in Tra Vinh
Factors affecting undergraduate students’ motivation at a university in Tra VinhFactors affecting undergraduate students’ motivation at a university in Tra Vinh
Factors affecting undergraduate students’ motivation at a university in Tra Vinh
AJHSSR Journal
 
Maximize Your Twitch Potential!..........
Maximize Your Twitch Potential!..........Maximize Your Twitch Potential!..........
Maximize Your Twitch Potential!..........
SocioCosmos
 

Recently uploaded (10)

STUDY ON THE DEVELOPMENT STRATEGY OF HUZHOU TOURISM
STUDY ON THE DEVELOPMENT STRATEGY OF HUZHOU TOURISMSTUDY ON THE DEVELOPMENT STRATEGY OF HUZHOU TOURISM
STUDY ON THE DEVELOPMENT STRATEGY OF HUZHOU TOURISM
 
UR BHATTI ACADEMY AND ONLINE COURSES.pdf
UR BHATTI ACADEMY AND ONLINE COURSES.pdfUR BHATTI ACADEMY AND ONLINE COURSES.pdf
UR BHATTI ACADEMY AND ONLINE COURSES.pdf
 
快速办理(worcester毕业证书)伍斯特大学毕业证PDF成绩单一模一样
快速办理(worcester毕业证书)伍斯特大学毕业证PDF成绩单一模一样快速办理(worcester毕业证书)伍斯特大学毕业证PDF成绩单一模一样
快速办理(worcester毕业证书)伍斯特大学毕业证PDF成绩单一模一样
 
Dominate Reddit Discussions.............
Dominate Reddit Discussions.............Dominate Reddit Discussions.............
Dominate Reddit Discussions.............
 
一比一原版(AU毕业证)英国阿伯丁大学毕业证如何办理
一比一原版(AU毕业证)英国阿伯丁大学毕业证如何办理一比一原版(AU毕业证)英国阿伯丁大学毕业证如何办理
一比一原版(AU毕业证)英国阿伯丁大学毕业证如何办理
 
原版制作(Hull毕业证书)赫尔大学毕业证Offer一模一样
原版制作(Hull毕业证书)赫尔大学毕业证Offer一模一样原版制作(Hull毕业证书)赫尔大学毕业证Offer一模一样
原版制作(Hull毕业证书)赫尔大学毕业证Offer一模一样
 
HMS Facebook Stories All V1 06092024.docx
HMS Facebook Stories All V1 06092024.docxHMS Facebook Stories All V1 06092024.docx
HMS Facebook Stories All V1 06092024.docx
 
The Impact of Work Stress and Digital Literacy on Employee Performance at PT ...
The Impact of Work Stress and Digital Literacy on Employee Performance at PT ...The Impact of Work Stress and Digital Literacy on Employee Performance at PT ...
The Impact of Work Stress and Digital Literacy on Employee Performance at PT ...
 
Factors affecting undergraduate students’ motivation at a university in Tra Vinh
Factors affecting undergraduate students’ motivation at a university in Tra VinhFactors affecting undergraduate students’ motivation at a university in Tra Vinh
Factors affecting undergraduate students’ motivation at a university in Tra Vinh
 
Maximize Your Twitch Potential!..........
Maximize Your Twitch Potential!..........Maximize Your Twitch Potential!..........
Maximize Your Twitch Potential!..........
 

FFMEET: running a non-profit conference system

  • 1. ffmuc: FFMEET running a non-profit conference system 3 months later
  • 2. awlnx ● Annika Wickert ● Senior Network Engineer @AS51324 ● Twitter @awlnx Krombel ● Matthias Kesler ● IT Consultant ● Twitter @kr0mbel 2 Who are we?
  • 3. 3 Jitsi to close the social gap during corona • An upcoming Freifunk Meeting was about to get canceled because of Corona • In the past we used NextCloud talk but we had issues (max 6. users per conference) ■ We needed something different for at least 20 - 30 people ■ jitsi seemed easy and straightforward to install ● “apt install jitsi-meet” => Done ¯_(ツ)_/¯
  • 4. 4 Debian package wins over Docker • We had some issues with docker-proxy in the past • docker image didn’t look up to date • With scaling in mind ■ Planned using dedicated hardware for Jitsi anyway ■ Debian packages are available
  • 5. 5 First test - looks promising • Testing with some people of Freifunk Munich • Worked like a charm, with a handful of people • The idea was born: ■ Maybe other people have the same problem? ■ Why not open it for the public? ■ Maybe teachers, healthcare workers, etc. need that, too?
  • 6. 6 We need insights! Let’s monitor! • How many users are on the platform? • What’s the impact on the Hardware/VMs? • Do we have bottlenecks? If so: Where?
  • 7. 7 What we needed for success • First servers got donated by individuals (max 4 cores) ■ Went well as long as we only had small conferences • Schools started to use our infrastructure => cheap servers were not enough anymore → Videobridges need powerful hardware and we may need many of them! → Looking for sponsors
  • 9. 9 The project gains attention • Interviews • BR, DigitalCourage, Focus Online, Stimme, SZ, … • diverse Listen
  • 10. 10 Problems start - Prosody • Change network_backend to “epoll” - No more 1024 connections as a limit
  • 11. 11 Don’t tune only one part of the stack - nginx • 502 all over the place • Raise the number of nginx workers • Raise max open files limit
  • 12. 12 How do we update without user impact? • We can upgrade the videobridges one by one • We query the videobridge API and wait for 0 users then run our custom upgrade script • Upgrades of videobridges just happen in the background during the day/night • Jicofo/Prosody upgrades … sadly we have to do them manually but we don’t need to update them very often
  • 13. 13 Holidays are over … school starts - 20.04.2020 • All time record before was around 600 Users
  • 14. 14 21.04.2020 - Another record! Sadly also another bug ● 20 min. to find the issue, isolate and fix it! ● Unfortunately there was an other bug hiding after the crash ● We run at reduced capacity for a few hours but as we have so many servers that was not an issue. ● Crash of prosody crashed some important DBs :/ ⇒ Loadbalancing failed
  • 15. 15 New records every day! • On 29.04.2020 we hit another highscore of 1503 concurrent users. Servers had no issues!
  • 16. 16 People tend to be in firewalled networks => TURN • To connect devices which cannot properly transmit data to the videobridge (UDP blocked) • videobridge2 used to provide a TCP Fallback to 443 … but it’s not real HTTPS traffic so NGFW tend to block it • Solution TURN (Traversal Using Relays around NAT) RFC5766 and 6156 • Ugly “Feature” in browsers => They prefer the first turnserver in the list … => Patch Jitsi to randomize the sequence
  • 17. 17 People ask for bigger meetings • Octo solves the problem of only one videobridge per conference (server-side is no longer an issue) • Sadly video traffic is unencrypted between videobridges => we need a solution for that • As we do high packet rates we need a full-mesh VPN without a central point where all traffic has to pass =>
  • 18. 18 82 people in one meeting and nebula in action
  • 19. 19 Octo on all bridges
  • 20. 20 Bugs with Octo • Videobridges join XMPP via pubsub and MUC => no region info in pubsub • Update to jitsi-videobridge2 unstable
  • 21. 21 We are running on the internet aren’t we? • From time to time there are issues in the internet beyond our control (route leaks, connectivity issues between providers, maintenance) • We have many servers and providers so we can easily mitigate connectivity issues
  • 22. 22 We got more popular so DDOS attacks happen • People attacked our infrastructure and didn’t stop when they couldn’t bring down #FFMEET • They started to attack other parts of our infrastructure
  • 23. 23 Custom code build on upstream/master • We no longer run the upstream packages for most software in the stack • We run a fork from latest master with some additions • jitsi-videobridge2 is still the latest unstable package on all hosts • We forked the electron app
  • 24. 24 Our infrastructure 3 months ago!
  • 26. 26 Some figures? 3 Months of #FFMEET • ~ 810TB videotraffic • ~ 847.955.871 HTTPS Requests • ~ 360 CPU Cores / 720GB of RAM • ~ 800.000 Unique Users • ~ 500 tickets solved (many schools, charity projects, individuals etc.) • ~ 200 Euro / day • 2 Admins and at least 15 people in the background for translations, press, wiki etc
  • 27. 27 Some unexpected things happen • You get famous in Peru and they use #FFMEET as their primary conf system • Schools start to depend on you • Small companies start to depend on you
  • 28. 28 What’s next? • Hopefully more happy users! • Establishing a long term roadmap for the service • Gathering money and resources to secure the funding of the project even more • Every penny helps! paypal.me/ffmucspenden
  • 29. 29 Thank you very much! A special thanks to all who helped writing/translating the FAQs. Also thank you to all the users and of course our supporters:
  • 30. 30 Firefox breaks user experience • Users experience problems especially in large conferences (25-40 people) • Add warning for Firefox users … it’s mostly ignored … so problems still exists • Deactivate Firefox support and release FreifunkMeet app →https://github.com/freifunkMUC/jitsi-meet-electron/releases
  • 31. 31 DoH / DoT Public Resolver • As DNS is an unencrypted protocol there are approaches to develop a more secure one • We offer DoH (native in Firefox) / DoT (native since Android 9) / DNSCrypt