Hany Fahim from VM Farms presents How brazil kept us up at night tales from the ops side
•Download as PPTX, PDF•
0 likes•422 views
Hany recounts the story of what happened when Brazil took down his network, what happened, why it happened and what you can learn from it. Presented at Tech Toronto Meetup March 2016 For videos of this presentation and more, visit our channel at: https://www.youtube.com/c/techtorontoorg Follow us on Twitter: https://twitter.com/techtorontoorg Facebook: https://www.facebook.com/techtorontoorg Instagram: https://www.instagram.com/techtorontoorg/
Recommended
More Related Content
What's hot
What's hot (9)
Similar to Hany Fahim from VM Farms presents How brazil kept us up at night tales from the ops side
Similar to Hany Fahim from VM Farms presents How brazil kept us up at night tales from the ops side (20)
More from TechTO
More from TechTO (20)
Recently uploaded
Recently uploaded (20)
Hany Fahim from VM Farms presents How brazil kept us up at night tales from the ops side
- 1. How Brazil kept us up at night Tales from the Ops side By Hany Fahim Founder and CEO @iHandroid @vmfarms
- 2. Wednesday, December 16th, 2015 8:00pm EST
- 3. Customer down! One of our customers had gone down and they were getting a flood of alerts. Jumped online to begin investigation. Overload alerts from all their API servers. Massive spike in traffic!
- 4. Was this an attack? Our immediate thought was some form of DDoS. Customer is in the security and VPN space. Started analyzing traffic, looking for patterns. Picked up the phone to contact the customer...
- 5. Another customer down! Another customer went down! Also in the security/VPN space. Same symptoms: overloaded servers, big spike in traffic. This was quite odd. 2 different customers, same space. Was there some event going on?
- 6. Brazilian IPs After analyzing the traffic, the vast majority of IPs were coming from Brazil. Still not convinced it wasn't an attack of sorts. Should we defend or scale? Contacted each customer, confirmed spikes in signups and usage. What would trigger a spike in security and VPN services?
- 7. Took to Twitter Searched “brazil”, immediately there were a flood of tweets: “BIG BROTHER en ACCION en Brazil!!! Justicia ordena bloquear WhatsApp durante 48 horas en Brasil” BIG BROTHER in action in Brazil!!! Justice ordered block WhatsApp for 48 hours in Brazil “El Gobierno de #Brazil ordenó bloquear #WhatsApp durante dos días!! Creo que muchos estarán en la carcel el fin de semana.” The government ordered #Brazil #WhatsApp block for two days!! I think many will be in jail over the weekend.
- 8. It was true Brazilian telecom companies were angry at their diminishing profits as more and more users communicate over WhatsApp. Apparently 93% of Brazil's internet population uses WhatsApp. Doctors use it to communicate with their patients. Businessmen use it to conduct transactions. People who are too poor to afford a cell phone have embraced its free services. With an Internet population of 100 million, that's 93 million users! (50% of the entire population of Brazil). It is the single most used app in the country.
- 10. PC Conectado In 2003, the Brazilian government launched an initiative to offer low-cost tax-free computers to anyone who wanted it. They mandated the use of Linux and Open Source Software, and outright rejected Microsoft's bid for OS of choice. This included all government ministries and state-owned systems. This move was widely publicized in the media.
- 11. Think about this for a minute.
- 12. Linux and OSS Linux has come a long way, but it still requires some technical know-how to operate. Unlike Windows and Mac, you usually have to pop open the hood and tinker. After more than a decade of this program, the result is a highly technical population.
- 13. So let’s see... Government enables the people by giving them technical knowledge, then tries to block access to the single most used app in the country. No wonder these VPN services were getting hit hard!
- 14. Time to scale up and out! Threw everything we had. Server after server, traffic kept soaring. By about midnight, things plateaued and sites were stable again.
- 15. Or so we thought... Around 5:00am EST, traffic spiked again, this time much higher than the previous peak. Brazil was waking up! One customer was already 5x their original size. Components we thought would never break started to buckle.
- 16. Then at 10:30am EST everything stopped.
- 17. Traffic calmed Load subsided, traffic waned, alerts cleared. What just happened? Took to Twitter: The ban had been lifted! The people won!
- 18. Lessons learned 1. Government censorship and blockades rarely works, especially when you arm your people with knowledge. 2. We are way more connected than you think. 3. Twitter is an invaluable resource for finding out what’s happening RIGHT NOW.
Editor's Notes
- We at VM Farms have observed this directly. We've expanded our own hiring practices to look for candidates globally, and there is a large % of highly qualified candidates from Brazil that regularly apply.
- Being in the security space, encryption was a key component, and encryption is not free. SSL termination points started maxing out their capacity. Not only did we need to scale the customer's app, we had to deal with CPU overload due to all this encryption.