Alexander McMillen, profile picture
Uploaded byAlexander McMillen
PPTX, PDF111 views

@HOPENOC 2016 Network Overview

This document summarizes the network upgrades and operations for HOPENOC 2016. Key points include: installing 10G fiber connections and running fiber between floors for redundancy; sourcing and configuring new networking hardware including switches, wireless access points, and controllers; obtaining IP address space and establishing BGP; transporting and staging equipment; and standing up critical network services just before the conference. Network usage peaked at over 500Mbps wireless usage and 1.76Gbps total bandwidth. Monitoring showed high wireless client counts but only moderate issues. Lessons learned include encouraging more encrypted WiFi usage and bandwidth consumption.

Embed presentation

Download to read offline
@HOPENO PUSH MORE BANDWIDTHFri Sat Sun
OUR START….
@HOPENOC 2016 SOME THINGS STAYED THE SAME… ▸Ordered 10G point-to-point to 111 8th Avenue ▸Carrier had to re-splice the fiber at the Hotel Penn… :( ▸Ordered cross connect at Equinix to Hurricane Electric. ▸Turned up a 10G port with Hurricane Electric. ▸Yellow Fiber Networks provided a Brocade XMR 4000 for the core. ▸Had to run a TON of fiber optic cabling….
@HOPENOC 2016 SOME THINGS CHANGED…. ▸After previous fiber cut between 1 & 18, we teamed up with Wave- 2-Wave.com to get 3 pairs of redundant fiber between the floors. ▸We ran 10G everywhere! ▸3 pair from Mezz to 18 ▸3 pair from Mezz to 6 ▸1 pair between switches on 18 ▸1 pair for Mezz to DEMARC
@HOPENOC 2016 HARDWARE SOURCING… ▸We had some turnover at our previous sponsor for wireless and distribution hardware, and had to find a new one… ▸As it turns out, sourcing 50 WAPs alone is difficult, let alone… that plus… ▸10 POE switches ▸Capable wireless controller ▸Luckily, we found an AWESOME partner at Aruba Networks, which provided the POE switches and controllers.
@HOPENOC 2016 HARDWARE SOURCING… ▸We received 10 shiny new switches straight from HPE Discover conference in Las Vegas. ▸Alex picked up 50 Wireless Access Points and 2 Controllers (redundancy!) from Aruba in Washington DC. ▸But… there were some additional challenges…
@HOPENOC 2016 HARDWARE SOURCING… ▸Where was all this going to go? ▸Roka Security provided space for initial hardware staging and sparing some of Alex’s time for configuration…..
@HOPENOC 2016 BUT WAAIT, THERE’S MORE… ▸Those shiny new switches… ▸Were so shiny and new they didn’t have the 10G cards or optics in-stock… and they were back ordered until September… ▸Luckily, Aruba Networks is such an awesome partner they sent an additional 8 previous generation switches, 10G cards and optics OVERNIGHT from California.
@HOPENOC 2016 AND HOW ABOUT IP SPACE? ▸IPv4 depletion is a real problem. ARIN is out, and RIPE will be soon, but AFRINIC has some… ▸We thought about becoming an African ISP, but as it turns out that’s expensive and time consuming. ▸We reached out to Prefix Broker, a broker that sources available IPs, and they were nice enough to hook us up with… ▸/19 of IPv4 (registered to A&F Networks in Amsterdam) ▸/29 of IPv6 (registered to Falco Networks)
@HOPENOC 2016 AND REMEMBER HOW WE WANTED TO BECOME AN ISP? ▸Hurricane Electric hooked us up with a 4 digit ASN! ▸We are AS6427. ▸We turned up IPv4 and IPv6 BGP sessions and originated our IP assignments from Prefix Broker.
@HOPENOC 2016 BUT, HOW WOULD ALL THE GEAR GET TO NYC?▸At the end of the day, we had 4 pallets of equipment to bring to HOPE. ▸MAGFest was kind enough to allow us to break down our pallets and do some last-minute configuration. ▸We drove up two cars of gear with Radio Statler.
BUT WHAT ABOUT
@HOPENOC 2016 WE SPENT THE LAST WEEK CONFIGURING NETWORK DEVICES
@HOPENOC 2016 AS A RESULT OF OUR PREPARATION, THE CORE WAS UP IN UNDER AN HOUR…▸Before there were even tables….
@HOPENOC 2016 SO WE JUST LEFT IT ON THE FLOOR…. ▸And improved the cooling for the core…
AND THEN THERE’S THE HARD
@HOPENOC 2016 ALL THE CABLES! ▸there’s a lot!!! it may not (all) be pretty, but it works!
@HOPENOC 2016 REMEMBER THAT WAVE-2-WAVE FIBER WE TALKED ABOUT?▸It kicks ass, and we ran 600 meters of it…
@HOPENOC 2016 BUT WHAT ABOUT THOSE PESKY SERVICES WE NEED… ▸DHCP server for IPv4 and IPv6 ▸Firewall/NAT for press, speaker, and management access ▸Monitoring ▸Bandwidth and system health ▸Internal and external ping
@HOPENOC 2016 DUE TO HARDWARE ISSUES, WE HAD TO GET THOSE UP LAST MINUTE…▸On a Windows 10 laptop running Hyper-V….. :( ▸Sorry, RMS… but, all the actual services ran on GNU/Linux.
@HOPENOC 2016 WE HAD SOME AMAZING MONITORING IN PLACE…▸We utilized Observium for bandwidth and health monitoring for systems. ▸Panopta was utilized for Internal and External availability monitoring. ▸They checked our external availability every 60 seconds from 3 different geographical areas. ▸All of our deployed wireless access points, controllers, switches, and core were monitored via an Internal sensor appliance. ▸Notifications were sent automagically to Slack for rapid response by the NOC team.
@HOPENOC 2016 WE HAD SOME AMAZING MONITORING IN PLACE… ▸We launched a public status website, which included real-time W
SOME STATS…
@HOPENOC 2016 NETWORK CONSUMPTION ▸We achieved 99.94% overall network uptime. ▸Over 500 Mbps pushed by wireless clients at peak. ▸A “small” DDoS attack (potentially an insider threat) caused our overall peak at 1.76 Gbps. ▸Over 800 simultaneous wireless clients at peak. ▸Over 1300 active DHCP leases at peak. ▸There was a TOR relay node in the NOC NOC (with 16,507 IPv4 connections) ▸According to the Aruba Networks controller, only around 20 users experienced “degraded” WiFi quality at peak. ▸We walked around and confirmed it wasn’t actually degraded… :)
INTERESTI NG
@HOPENOC 2016 FINDINGS ▸Again, a lot of people wondered why: ▸They had a public IP in Amsterdam ▸Why they wouldn’t get 10 Gbps on their phone…. ▸Seriously, physics.. c’mon people…. ▸Most speed test sites couldn’t keep up… ▸You were more likely to have your system compromised by an external source than from attendees of HOPE. ▸Verified with Snort and Honeypots.
@HOPENOC 2016 FINDINGS ▸ Under 50 users at peak were using WPA2 Enterprise (seriously)? - it even had a valid Let’s Encrypt certificate… ▸ Over 80% of users were connected to unencrypted WiFi ▸ Most users were on 5 GHz ▸ FuckTheNoc user appeared again and we nullrouted him (thanks, ATG)… ▸ Radio Statler still loves us ;)
@HOPENOC 2016 AND REMEMBER THOSE 50 USERS ON WPA2 ENTERPRISE?▸ They’re safe, but not the other 740 wireless clients (at peak) using the unencrypted wireless. ▸ We found…. ▸ 680 pictures ▸ 65% were no good. ▸ 10 excel documents which contained passwords ▸ You guys spend a lot of time on Pokemon Go ▸ We should really ban that…
JUST KIDDING. <3 BUT NEXT TIME WE’RE RATE
NEXT TIME…
@HOPENOC 2016 NEXT TIME.. ▸We’re considering multi-homing with BGP. ▸Having users push more bandwidth ▸All of the live streams (audio and video) hosted locally… ▸You did NOT push enough bandwidth. ▸Reid from Hurricane Electric is VERY disappointed and may only give us 1 Gbps… COME ON PEOPLE. ▸Rate limiting unencrypted WiFi to 0.01 Mbps…. and maybe make a wall of sheep. ▸Potentially having to provide service for another floor…. ;)
WE COULD NOT HAVE DONE
@HOPENOC 2016 WITHOUT OUR AMAZING SPONSORS! ^^ Is hiring in VA/DC ^^
@HOPENOC 2016 AND @HOPENOC 2016 TEAM….

More Related Content

PDF
Core Bluetooth on iOS
bystorywithoutend
 
PDF
Penghargaan
byAgus Romadhon
 
DOCX
KHolder NEH BK Historical Essay
byKanene Ayo Holder
 
PPT
Trabajo De TecnologíA
byLucyla
 
PDF
Tournament blochure
byJaydeep Songara
 
PDF
Operational Excellence Tony Vines
byTony Vines
 
PDF
My Experience1
byHashan Travis Haputhanthri
 
PPTX
4.1 distribución CV travel open apps
byOfe Bepa
 
Core Bluetooth on iOS
bystorywithoutend
 
Penghargaan
byAgus Romadhon
 
KHolder NEH BK Historical Essay
byKanene Ayo Holder
 
Trabajo De TecnologíA
byLucyla
 
Tournament blochure
byJaydeep Songara
 
Operational Excellence Tony Vines
byTony Vines
 
My Experience1
byHashan Travis Haputhanthri
 
4.1 distribución CV travel open apps
byOfe Bepa
 

Similar to @HOPENOC 2016 Network Overview

PDF
HOPENOC Closing Ceremonies Presentation - Circle of HOPE
byAlexander McMillen
 
PPTX
liaison-2019-09-30-itu-t-tsag-ietf-iab-ls-on-new-ip-shaping-future-network-at...
byMohammadSwerki2
 
PPT
Werbach slides friday
byKevin Werbach
 
PDF
Data Networks: Next-Generation Optical Access toward 10 Gb/s Everywhere
byXi'an Jiaotong-Liverpool University
 
PDF
Photonics21 – Next-Generation Optical Internet Access: Roadmap for Broadband ...
byXi'an Jiaotong-Liverpool University
 
PPTX
Nokia Autonomous Broadband Network Seminar.pptx
byrmatos1
 
PPT
Impact on Society – the Light at the end of the Tunnel
byTal Lavian Ph.D.
 
PPTX
Wowmom panel-claudio-2013
byIIT CNR
 
PPT
Abundant Bandwidth and how it affects us
byTal Lavian Ph.D.
 
PDF
Infographic: Mobile Industry Leaders Provide a Glimpse of Our Future
byCisco Service Provider
 
PDF
Sharing is caring: Real-life shared infrastructure experience
byADVA
 
PDF
Capacity planning
byTelia Carrier
 
PPT
Supernova 2009 Overview and Resource Guide
byTerrorNova Guild
 
PPT
Services and applications’ infrastructure for agile optical networks
byTal Lavian Ph.D.
 
PPT
B Huiszoon Vederprijs2008 V Oct2009
bybashuiszoon
 
DOC
Noticias tel dic
byFrancisco Apablaza
 
PPTX
ARIN 34 IPv6 IAB/IETF Activities Report
byARIN
 
PPTX
Mr. Ottmar Krauss' presentation at QITCOM 2011
byQITCOM
 
PPTX
Astricon - Realities of Global Infrastructure in the Cloud
byCory von Wallenstein
 
PDF
Google Wave: Ripple or Tsunami for Research
byCameron Neylon
 
HOPENOC Closing Ceremonies Presentation - Circle of HOPE
byAlexander McMillen
 
liaison-2019-09-30-itu-t-tsag-ietf-iab-ls-on-new-ip-shaping-future-network-at...
byMohammadSwerki2
 
Werbach slides friday
byKevin Werbach
 
Data Networks: Next-Generation Optical Access toward 10 Gb/s Everywhere
byXi'an Jiaotong-Liverpool University
 
Photonics21 – Next-Generation Optical Internet Access: Roadmap for Broadband ...
byXi'an Jiaotong-Liverpool University
 
Nokia Autonomous Broadband Network Seminar.pptx
byrmatos1
 
Impact on Society – the Light at the end of the Tunnel
byTal Lavian Ph.D.
 
Wowmom panel-claudio-2013
byIIT CNR
 
Abundant Bandwidth and how it affects us
byTal Lavian Ph.D.
 
Infographic: Mobile Industry Leaders Provide a Glimpse of Our Future
byCisco Service Provider
 
Sharing is caring: Real-life shared infrastructure experience
byADVA
 
Capacity planning
byTelia Carrier
 
Supernova 2009 Overview and Resource Guide
byTerrorNova Guild
 
Services and applications’ infrastructure for agile optical networks
byTal Lavian Ph.D.
 
B Huiszoon Vederprijs2008 V Oct2009
bybashuiszoon
 
Noticias tel dic
byFrancisco Apablaza
 
ARIN 34 IPv6 IAB/IETF Activities Report
byARIN
 
Mr. Ottmar Krauss' presentation at QITCOM 2011
byQITCOM
 
Astricon - Realities of Global Infrastructure in the Cloud
byCory von Wallenstein
 
Google Wave: Ripple or Tsunami for Research
byCameron Neylon
 

Recently uploaded

PDF
investor pitch - iapploud | independent music platform
byVenkat Subramanian
 
PPTX
Creating content that converts into leads.
bySEONutri
 
PDF
Batman Forever album sličice.pdf panini album
byStefanFilipovic9
 
PDF
automobili.pdf panini album slicice ......
byStefanFilipovic9
 
PDF
GTI Solution Overview.pdf useful for security solution
by33fastfast
 
PPTX
Social_Media_Good_or_Bad_B2Plus_Lesson.pptx
byJuliaRutkowska4
 
investor pitch - iapploud | independent music platform
byVenkat Subramanian
 
Creating content that converts into leads.
bySEONutri
 
Batman Forever album sličice.pdf panini album
byStefanFilipovic9
 
automobili.pdf panini album slicice ......
byStefanFilipovic9
 
GTI Solution Overview.pdf useful for security solution
by33fastfast
 
Social_Media_Good_or_Bad_B2Plus_Lesson.pptx
byJuliaRutkowska4
 

@HOPENOC 2016 Network Overview

  • 1.
  • 2.
  • 3.
    @HOPENOC 2016 SOME THINGSSTAYED THE SAME… ▸Ordered 10G point-to-point to 111 8th Avenue ▸Carrier had to re-splice the fiber at the Hotel Penn… :( ▸Ordered cross connect at Equinix to Hurricane Electric. ▸Turned up a 10G port with Hurricane Electric. ▸Yellow Fiber Networks provided a Brocade XMR 4000 for the core. ▸Had to run a TON of fiber optic cabling….
  • 4.
    @HOPENOC 2016 SOME THINGSCHANGED…. ▸After previous fiber cut between 1 & 18, we teamed up with Wave- 2-Wave.com to get 3 pairs of redundant fiber between the floors. ▸We ran 10G everywhere! ▸3 pair from Mezz to 18 ▸3 pair from Mezz to 6 ▸1 pair between switches on 18 ▸1 pair for Mezz to DEMARC
  • 5.
    @HOPENOC 2016 HARDWARE SOURCING… ▸Wehad some turnover at our previous sponsor for wireless and distribution hardware, and had to find a new one… ▸As it turns out, sourcing 50 WAPs alone is difficult, let alone… that plus… ▸10 POE switches ▸Capable wireless controller ▸Luckily, we found an AWESOME partner at Aruba Networks, which provided the POE switches and controllers.
  • 6.
    @HOPENOC 2016 HARDWARE SOURCING… ▸Wereceived 10 shiny new switches straight from HPE Discover conference in Las Vegas. ▸Alex picked up 50 Wireless Access Points and 2 Controllers (redundancy!) from Aruba in Washington DC. ▸But… there were some additional challenges…
  • 7.
    @HOPENOC 2016 HARDWARE SOURCING… ▸Wherewas all this going to go? ▸Roka Security provided space for initial hardware staging and sparing some of Alex’s time for configuration…..
  • 8.
    @HOPENOC 2016 BUT WAAIT,THERE’S MORE… ▸Those shiny new switches… ▸Were so shiny and new they didn’t have the 10G cards or optics in-stock… and they were back ordered until September… ▸Luckily, Aruba Networks is such an awesome partner they sent an additional 8 previous generation switches, 10G cards and optics OVERNIGHT from California.
  • 9.
    @HOPENOC 2016 AND HOWABOUT IP SPACE? ▸IPv4 depletion is a real problem. ARIN is out, and RIPE will be soon, but AFRINIC has some… ▸We thought about becoming an African ISP, but as it turns out that’s expensive and time consuming. ▸We reached out to Prefix Broker, a broker that sources available IPs, and they were nice enough to hook us up with… ▸/19 of IPv4 (registered to A&F Networks in Amsterdam) ▸/29 of IPv6 (registered to Falco Networks)
  • 10.
    @HOPENOC 2016 AND REMEMBERHOW WE WANTED TO BECOME AN ISP? ▸Hurricane Electric hooked us up with a 4 digit ASN! ▸We are AS6427. ▸We turned up IPv4 and IPv6 BGP sessions and originated our IP assignments from Prefix Broker.
  • 11.
    @HOPENOC 2016 BUT, HOWWOULD ALL THE GEAR GET TO NYC?▸At the end of the day, we had 4 pallets of equipment to bring to HOPE. ▸MAGFest was kind enough to allow us to break down our pallets and do some last-minute configuration. ▸We drove up two cars of gear with Radio Statler.
  • 12.
  • 13.
    @HOPENOC 2016 WE SPENTTHE LAST WEEK CONFIGURING NETWORK DEVICES
  • 14.
    @HOPENOC 2016 AS ARESULT OF OUR PREPARATION, THE CORE WAS UP IN UNDER AN HOUR…▸Before there were even tables….
  • 15.
    @HOPENOC 2016 SO WEJUST LEFT IT ON THE FLOOR…. ▸And improved the cooling for the core…
  • 16.
  • 17.
    @HOPENOC 2016 ALL THECABLES! ▸there’s a lot!!! it may not (all) be pretty, but it works!
  • 18.
    @HOPENOC 2016 REMEMBER THATWAVE-2-WAVE FIBER WE TALKED ABOUT?▸It kicks ass, and we ran 600 meters of it…
  • 19.
    @HOPENOC 2016 BUT WHATABOUT THOSE PESKY SERVICES WE NEED… ▸DHCP server for IPv4 and IPv6 ▸Firewall/NAT for press, speaker, and management access ▸Monitoring ▸Bandwidth and system health ▸Internal and external ping
  • 20.
    @HOPENOC 2016 DUE TOHARDWARE ISSUES, WE HAD TO GET THOSE UP LAST MINUTE…▸On a Windows 10 laptop running Hyper-V….. :( ▸Sorry, RMS… but, all the actual services ran on GNU/Linux.
  • 21.
    @HOPENOC 2016 WE HADSOME AMAZING MONITORING IN PLACE…▸We utilized Observium for bandwidth and health monitoring for systems. ▸Panopta was utilized for Internal and External availability monitoring. ▸They checked our external availability every 60 seconds from 3 different geographical areas. ▸All of our deployed wireless access points, controllers, switches, and core were monitored via an Internal sensor appliance. ▸Notifications were sent automagically to Slack for rapid response by the NOC team.
  • 22.
    @HOPENOC 2016 WE HADSOME AMAZING MONITORING IN PLACE… ▸We launched a public status website, which included real-time W
  • 23.
  • 24.
    @HOPENOC 2016 NETWORK CONSUMPTION ▸Weachieved 99.94% overall network uptime. ▸Over 500 Mbps pushed by wireless clients at peak. ▸A “small” DDoS attack (potentially an insider threat) caused our overall peak at 1.76 Gbps. ▸Over 800 simultaneous wireless clients at peak. ▸Over 1300 active DHCP leases at peak. ▸There was a TOR relay node in the NOC NOC (with 16,507 IPv4 connections) ▸According to the Aruba Networks controller, only around 20 users experienced “degraded” WiFi quality at peak. ▸We walked around and confirmed it wasn’t actually degraded… :)
  • 25.
  • 26.
    @HOPENOC 2016 FINDINGS ▸Again, alot of people wondered why: ▸They had a public IP in Amsterdam ▸Why they wouldn’t get 10 Gbps on their phone…. ▸Seriously, physics.. c’mon people…. ▸Most speed test sites couldn’t keep up… ▸You were more likely to have your system compromised by an external source than from attendees of HOPE. ▸Verified with Snort and Honeypots.
  • 27.
    @HOPENOC 2016 FINDINGS ▸ Under50 users at peak were using WPA2 Enterprise (seriously)? - it even had a valid Let’s Encrypt certificate… ▸ Over 80% of users were connected to unencrypted WiFi ▸ Most users were on 5 GHz ▸ FuckTheNoc user appeared again and we nullrouted him (thanks, ATG)… ▸ Radio Statler still loves us ;)
  • 28.
    @HOPENOC 2016 AND REMEMBERTHOSE 50 USERS ON WPA2 ENTERPRISE?▸ They’re safe, but not the other 740 wireless clients (at peak) using the unencrypted wireless. ▸ We found…. ▸ 680 pictures ▸ 65% were no good. ▸ 10 excel documents which contained passwords ▸ You guys spend a lot of time on Pokemon Go ▸ We should really ban that…
  • 29.
    JUST KIDDING. <3 BUT NEXTTIME WE’RE RATE
  • 30.
  • 31.
    @HOPENOC 2016 NEXT TIME.. ▸We’reconsidering multi-homing with BGP. ▸Having users push more bandwidth ▸All of the live streams (audio and video) hosted locally… ▸You did NOT push enough bandwidth. ▸Reid from Hurricane Electric is VERY disappointed and may only give us 1 Gbps… COME ON PEOPLE. ▸Rate limiting unencrypted WiFi to 0.01 Mbps…. and maybe make a wall of sheep. ▸Potentially having to provide service for another floor…. ;)
  • 32.
  • 33.
    @HOPENOC 2016 WITHOUT OURAMAZING SPONSORS! ^^ Is hiring in VA/DC ^^
  • 34.