Scaling an Extreme Temporary Event Network for Burning Man

Burning Man 
Scaling an extreme 
temporary event 
network 
Matt Peterson 
matt@burningman.com 
Matt Peterson
Terry Ratcliff 
Reuters/Jim Urquhart 
Black Rock City, NV 
duncan.co
2 01 4 B ur n i ng M an IP N etwo r k 
Team I.T.S. is Backbone, Camera Girl, Cat, Domo, Huckleberry, 
Little Meat, MattStep, Mushroom, Prof. Fox, PornStar, Ralf, 
Reset, Sawdust, Spank Me, Taz, Whiskey Devil, and Wild Card 
gw-noc 
lo0 - 162.212.145.252 
Primary 
Cost 100 
Tertiary 
Cost 400 
Secondary 
Cost 150 
Channel 5 
TX 11.385GHz 
RX 10.895GHz 
gw-depot 
lo0 - 162.212.145.254 
Channel 2 
TX 11.265GHz 
RX 10.775GHz 
Secondary 
Cost 150 
Channel 150 
5.750GHz 
Channel 140 
TX 5.700GHz 
RX 5.750GHz 
dw-tower-noc 
port1 - 162.212.144.26 
dw-noc-tower 
port1 - 162.212.144.27 
High Desert 
Internet 
dw-tower-depot 
port1 - 162.212.144.10 
dw-depot-tower 
port1 - 162.212.144.11 
dw-noc-depot 
port1 - 162.212.144.75 
dw-depot-noc 
port1 - 162.212.144.74 
ap-tower 
br0.2 - 162.212.144.2 
st-noc-tower 
br0.6 - 162.212.144.34 
Poito Eagle Ridge 
st-depot-tower 
br0.4 - 162.212.144.18 
ge0/3 - 162.212.144.25 ge0/2 - 162.212.144.9 
Primary 
Cost 100 
Tertiary 
Cost 400 
m a t t @ b u r n i n g m a n . c o m 2 0 1 4 - 1 0 - 0 2 1 8 : 1 0 
Channel 5 
TX 10.895GHz 
RX 11.385GHz 
Channel 2 
TX 10.775GHz 
RX 11.265GHz 
Channel 1 
TX 17.765GHz 
RX 19.325GHz 
Channel 1 
TX 19.325GHz 
RX 17.765GHz 
sw-noc2 
vl7 - 162.212.144.123 
sw-noc1 
vl7 - 162.212.144.122 
v94 - NOC LAN 
v111 - First Camp 
v114 - Media Mecca 
v115 - NOC Inside 
v121 - Webcast 
v94 - NOC LAN 
v103 - Artery 
v210 - Digerati & Devas 
ap-noc2 
br0.25 - 162.212.144.146 
ap-noc10 
br0.25 - 162.212.144.149 
ap-noc7 
br0.25 - 162.212.144.148 
sw-depot 
vl8 - 162.212.144.130 
Channel 6 
2.437GHz 
apb-noc-local 
br0 - 100.96.94.2 
Channel 1 
2.412GHz 
apb-depot-local 
br0 - 100.96.91.2 
Channel 1 
2.412GHz 
apb-noc1-inside 
br0 - 100.96.115.2 
v122 - OC1/West Wing 
Radio Frequency 
UTP Ethernet 
Licensed 
Unlicensed 
OSPF weight 
Transit 
Channel 147 
5.735GHz 
Channel 153 
5.765GHz 
Channel 151 
5.755GHz 
Channel 157 
5.785GHz 
apa-noc 
br0.33 - 162.212.145.35 
Channel 165 
5.825GHz 
v91 - Depot, Logistics 
st-light-depot 
br0.17 - 162.212.144.210 
st-light-noc 
br0.27 - 162.212.144.202 
st-airport 
br0.101 - 100.96.101.2 
v107 - Commissary Office 
v113 - Laminates 
v125 - Big Office 
v126 - Human Resources 
v127 - Container Office 
gw-light 
lo0 - 162.212.145.248 
v100 - Accounting 
v197 - Commissary Public 
v215 - GPE Camp 
v221 - Ticketfly 
v224 - Boob 
sw-commissary 
mgmt.34 - 162.212.146.142 
ap-depot-omni 
br0.10 - 162.212.144.66 
Channel 163 
5.815GHz 
Channel 11 
2.462GHz 
apb-commissary-local 
br0 - 100.96.107.2 
st-poopdudes 
br0.216 - 100.96.216.2 
st-quad4-depot 
br0.15 - 162.212.144.106 
st-quad4-noc 
br0.23 - 162.212.144.170 
v104 - BMIR 
v108 - DMV 
v120 - Ranger HQ 
gw-quad4 
lo0 - 162.212.145.249 
v118 - Playa Info 
v120 - Ranger HQ 
sw-quad4 
mgmt.35 - 162.212.146.130 
Channel 6 
2.437GHz 
apb-quad4-local 
br0 - 100.96.95.2 
st-cafe 
br0.216 - 100.96.216.2 
Channel 11 
2.462GHz 
apb-cafe-local 
br0 - 100.96.106.3 
st-pgepoint1 
br0.117 - 100.96.117.2 
Channel 6 
2.437GHz 
apb-pgepoint1 
br0 - 100.96.117.4 
st-ghetto 
br0.211 - 100.96.211.2 
Channel 1 
2.412GHz 
apb-ghetto-local 
br0 - 100.96.211.3 
br0.22 - 162.212.144.162 
gw-esd 
st-esd-depot 
br0.14 - 162.212.144.98 
st-esd-noc 
lo0 - 162.212.145.253 
v20 - ESD Comm. 
v111 - Incident Comm. Post 
st-box-depot 
br0.13 - 162.212.144.90 
st-box-noc 
br0.21 - 162.212.144.154 
gw-box 
lo0 - 162.212.145.250 
v105 - Box Office 
v116 - GPE 
st-power-depot 
br0.18 - 162.212.144.218 
st-power-noc 
br0.28 - 162.212.144.226 
gw-power 
lo0 - 162.212.145.247 
v119 - Power Camp 
Channel 11 
2.462GHz 
apb-power-local 
br0 - 100.96.97.2 
st-heavy-depot 
br0.16 - 162.212.144.114 
st-heavy-noc 
br0.24 - 162.212.144.178 
gw-heavy 
lo0 - 162.212.145.251 
v96 - Heavy Camp 
v109 - HGH "Rampart" 
v112 - Heavy Office 
Channel 6 
2.437GHz 
apb-heavy-local 
br0 - 100.96.96.2 
st-esdstation9 
br0.130 - 100.96.130.2 
st-esdstation3 
br0.129 - 100.96.129.2 
ap-noc5 
br0.25 - 162.212.144.147 
ap-depot-sector 
br0.10 - 162.212.144.67 
Channel 153 
5.795GHz 
Channel 6 
2.437GHz 
apb-depot-dispatch 
br0 - 100.96.91.3
Where We Started 
• The network worked, but it wasn’t easy 
– Large L2 bridged architecture, minimal L3 
segmentation, multiple NAT layers 
– Two distinct “business units” 
– Manual configuration, “tribal knowledge” 
– Numerous single points of failure
Where We Went 
• Needed to operate as a unified team 
– Consistent support experience, improved RF 
spectral efficiency, coordinated IP allocations 
• Standardized COTS equipment 
– “CCIE off the street” factor, escalation path 
• Standardized service offerings 
– Org department handoff’s always wired gigE; as 
aggregated “islands” or single demarc 
– Participant camps supply very prescriptive 
equipment, “self-install” provisioning
Where We Went 
• Route, always 
– No L2 segments past a single device 
– OSPF everywhere, core backbone & “islands” 
– Segment where possible, even over WiFi 
• Automation 
– Initially covering all routers & switches 
– Target goal to cover any device with a config or 
supplemental service (DNS, monitoring)
Automation! 
• Held bakeoff (mid 2011 evaluation) 
– Homegrown YAML  config templates 
– Prototyped NCG (see NANOG49 Tutorial 
“Automating Network Configuration”) 
• NCG won (3 yrs ago) 
– Open source, vendor agnostic 
– Initial steep curve, very easy to embrace 
– Principal developer already a team member
Actual git example
Scaling an Extreme Temporary Event Network for Burning Man
Summary Overview 
• {Automation} data modeling isn’t easy 
– Imagine all your inputs & outputs (device configs, 
DNS, monitoring, billing, etc.) 
• Single source of truth 
– Git, a wiki, fancy IPAM: choose what fits your 
organization’s workflow, stress level, & budget 
• Start at L8 and L1, meet in the middle 
– People + physical layer = organic processes 
– End goal is to be efficient, not become a SW dev
More.. unconference? 
Matt Peterson 
matt@burningman.com
1 of 11

Recommended

Au ama 10.15.14 by
Au ama 10.15.14Au ama 10.15.14
Au ama 10.15.14rrandall1
179 views30 slides
Soalan kt tahun 5 pksr 2 1 2013 by
Soalan kt tahun 5 pksr 2 1 2013Soalan kt tahun 5 pksr 2 1 2013
Soalan kt tahun 5 pksr 2 1 2013pang5gabanz
439 views8 slides
Analysis of previous students work by
Analysis of previous students workAnalysis of previous students work
Analysis of previous students workJudiAbdi
181 views5 slides
Tutela giurisdizionale dei "rifugiati" (How to ensure judicial enforcement of... by
Tutela giurisdizionale dei "rifugiati" (How to ensure judicial enforcement of...Tutela giurisdizionale dei "rifugiati" (How to ensure judicial enforcement of...
Tutela giurisdizionale dei "rifugiati" (How to ensure judicial enforcement of...Nicola Canestrini
736 views16 slides
"Assolti. Ma forse sonocolpevoli". ("Acquitted. But they may are guilty") by
"Assolti. Ma forse sonocolpevoli". ("Acquitted. But they may are guilty")"Assolti. Ma forse sonocolpevoli". ("Acquitted. But they may are guilty")
"Assolti. Ma forse sonocolpevoli". ("Acquitted. But they may are guilty")Nicola Canestrini
697 views23 slides
Kerja kelompok 1 urutan 5 fisika by
Kerja kelompok 1 urutan 5 fisikaKerja kelompok 1 urutan 5 fisika
Kerja kelompok 1 urutan 5 fisikaImanuel SImatupang
1.2K views24 slides

More Related Content

Viewers also liked

AAKASH FINAL REPORT(1) by
AAKASH FINAL REPORT(1)AAKASH FINAL REPORT(1)
AAKASH FINAL REPORT(1)Aakash Asokan
495 views69 slides
Immigrazione. Questione di dignità. Immigration, a question of human dignity by
Immigrazione. Questione di dignità. Immigration, a question of human dignityImmigrazione. Questione di dignità. Immigration, a question of human dignity
Immigrazione. Questione di dignità. Immigration, a question of human dignityNicola Canestrini
289 views18 slides
Tabla final de sergio montoya 6b by
Tabla final  de sergio montoya 6bTabla final  de sergio montoya 6b
Tabla final de sergio montoya 6bSERGI2002
623 views8 slides
Prototype solution2 by
Prototype solution2Prototype solution2
Prototype solution2abukky52
251 views3 slides
maaustria-local by
maaustria-localmaaustria-local
maaustria-localMary Ann Austria
335 views6 slides
Loch Ness Monster by
Loch Ness MonsterLoch Ness Monster
Loch Ness Monsterssb21
350 views10 slides

Viewers also liked(7)

Immigrazione. Questione di dignità. Immigration, a question of human dignity by Nicola Canestrini
Immigrazione. Questione di dignità. Immigration, a question of human dignityImmigrazione. Questione di dignità. Immigration, a question of human dignity
Immigrazione. Questione di dignità. Immigration, a question of human dignity
Nicola Canestrini289 views
Tabla final de sergio montoya 6b by SERGI2002
Tabla final  de sergio montoya 6bTabla final  de sergio montoya 6b
Tabla final de sergio montoya 6b
SERGI2002623 views
Prototype solution2 by abukky52
Prototype solution2Prototype solution2
Prototype solution2
abukky52251 views
Loch Ness Monster by ssb21
Loch Ness MonsterLoch Ness Monster
Loch Ness Monster
ssb21350 views
Did planning presentation 14th october by Sarah Kirkby
Did planning presentation 14th october Did planning presentation 14th october
Did planning presentation 14th october
Sarah Kirkby1K views

More from DevOps4Networks

Operationalize All the Network Things by Lori MacVittie by
Operationalize All the Network Things by Lori MacVittieOperationalize All the Network Things by Lori MacVittie
Operationalize All the Network Things by Lori MacVittieDevOps4Networks
610 views31 slides
Keynote: DevOps 4 Networks by JR Rivers of Cumulus Networks by
Keynote: DevOps 4 Networks by JR Rivers of Cumulus NetworksKeynote: DevOps 4 Networks by JR Rivers of Cumulus Networks
Keynote: DevOps 4 Networks by JR Rivers of Cumulus NetworksDevOps4Networks
721 views8 slides
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam... by
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...DevOps4Networks
2.3K views34 slides
When DevOps and Networking Intersect by Brent Salisbury of socketplane.io by
When DevOps and Networking Intersect by Brent Salisbury of socketplane.ioWhen DevOps and Networking Intersect by Brent Salisbury of socketplane.io
When DevOps and Networking Intersect by Brent Salisbury of socketplane.ioDevOps4Networks
1.1K views39 slides
Infrastructure API Lightning Talk by Jeremy Pollard of box.com by
Infrastructure API Lightning Talk by Jeremy Pollard of box.comInfrastructure API Lightning Talk by Jeremy Pollard of box.com
Infrastructure API Lightning Talk by Jeremy Pollard of box.comDevOps4Networks
1.4K views42 slides
SDN for Hybrid Cloud by
SDN for Hybrid CloudSDN for Hybrid Cloud
SDN for Hybrid CloudDevOps4Networks
607 views7 slides

More from DevOps4Networks(7)

Operationalize All the Network Things by Lori MacVittie by DevOps4Networks
Operationalize All the Network Things by Lori MacVittieOperationalize All the Network Things by Lori MacVittie
Operationalize All the Network Things by Lori MacVittie
DevOps4Networks610 views
Keynote: DevOps 4 Networks by JR Rivers of Cumulus Networks by DevOps4Networks
Keynote: DevOps 4 Networks by JR Rivers of Cumulus NetworksKeynote: DevOps 4 Networks by JR Rivers of Cumulus Networks
Keynote: DevOps 4 Networks by JR Rivers of Cumulus Networks
DevOps4Networks721 views
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam... by DevOps4Networks
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...
DevOps4Networks2.3K views
When DevOps and Networking Intersect by Brent Salisbury of socketplane.io by DevOps4Networks
When DevOps and Networking Intersect by Brent Salisbury of socketplane.ioWhen DevOps and Networking Intersect by Brent Salisbury of socketplane.io
When DevOps and Networking Intersect by Brent Salisbury of socketplane.io
DevOps4Networks1.1K views
Infrastructure API Lightning Talk by Jeremy Pollard of box.com by DevOps4Networks
Infrastructure API Lightning Talk by Jeremy Pollard of box.comInfrastructure API Lightning Talk by Jeremy Pollard of box.com
Infrastructure API Lightning Talk by Jeremy Pollard of box.com
DevOps4Networks1.4K views

Recently uploaded

Voice Logger - Telephony Integration Solution at Aegis by
Voice Logger - Telephony Integration Solution at AegisVoice Logger - Telephony Integration Solution at Aegis
Voice Logger - Telephony Integration Solution at AegisNirmal Sharma
31 views1 slide
6g - REPORT.pdf by
6g - REPORT.pdf6g - REPORT.pdf
6g - REPORT.pdfLiveplex
10 views23 slides
STPI OctaNE CoE Brochure.pdf by
STPI OctaNE CoE Brochure.pdfSTPI OctaNE CoE Brochure.pdf
STPI OctaNE CoE Brochure.pdfmadhurjyapb
13 views1 slide
virtual reality.pptx by
virtual reality.pptxvirtual reality.pptx
virtual reality.pptxG036GaikwadSnehal
11 views15 slides
Igniting Next Level Productivity with AI-Infused Data Integration Workflows by
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
257 views86 slides
Lilypad @ Labweek, Istanbul, 2023.pdf by
Lilypad @ Labweek, Istanbul, 2023.pdfLilypad @ Labweek, Istanbul, 2023.pdf
Lilypad @ Labweek, Istanbul, 2023.pdfAlly339821
9 views45 slides

Recently uploaded(20)

Voice Logger - Telephony Integration Solution at Aegis by Nirmal Sharma
Voice Logger - Telephony Integration Solution at AegisVoice Logger - Telephony Integration Solution at Aegis
Voice Logger - Telephony Integration Solution at Aegis
Nirmal Sharma31 views
6g - REPORT.pdf by Liveplex
6g - REPORT.pdf6g - REPORT.pdf
6g - REPORT.pdf
Liveplex10 views
STPI OctaNE CoE Brochure.pdf by madhurjyapb
STPI OctaNE CoE Brochure.pdfSTPI OctaNE CoE Brochure.pdf
STPI OctaNE CoE Brochure.pdf
madhurjyapb13 views
Igniting Next Level Productivity with AI-Infused Data Integration Workflows by Safe Software
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software257 views
Lilypad @ Labweek, Istanbul, 2023.pdf by Ally339821
Lilypad @ Labweek, Istanbul, 2023.pdfLilypad @ Labweek, Istanbul, 2023.pdf
Lilypad @ Labweek, Istanbul, 2023.pdf
Ally3398219 views
Data-centric AI and the convergence of data and model engineering: opportunit... by Paolo Missier
Data-centric AI and the convergence of data and model engineering:opportunit...Data-centric AI and the convergence of data and model engineering:opportunit...
Data-centric AI and the convergence of data and model engineering: opportunit...
Paolo Missier39 views
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors by sugiuralab
TouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective SensorsTouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective Sensors
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors
sugiuralab19 views
The details of description: Techniques, tips, and tangents on alternative tex... by BookNet Canada
The details of description: Techniques, tips, and tangents on alternative tex...The details of description: Techniques, tips, and tangents on alternative tex...
The details of description: Techniques, tips, and tangents on alternative tex...
BookNet Canada126 views
Unit 1_Lecture 2_Physical Design of IoT.pdf by StephenTec
Unit 1_Lecture 2_Physical Design of IoT.pdfUnit 1_Lecture 2_Physical Design of IoT.pdf
Unit 1_Lecture 2_Physical Design of IoT.pdf
StephenTec12 views
Case Study Copenhagen Energy and Business Central.pdf by Aitana
Case Study Copenhagen Energy and Business Central.pdfCase Study Copenhagen Energy and Business Central.pdf
Case Study Copenhagen Energy and Business Central.pdf
Aitana16 views
handbook for web 3 adoption.pdf by Liveplex
handbook for web 3 adoption.pdfhandbook for web 3 adoption.pdf
handbook for web 3 adoption.pdf
Liveplex22 views
Empathic Computing: Delivering the Potential of the Metaverse by Mark Billinghurst
Empathic Computing: Delivering  the Potential of the MetaverseEmpathic Computing: Delivering  the Potential of the Metaverse
Empathic Computing: Delivering the Potential of the Metaverse
Mark Billinghurst476 views
1st parposal presentation.pptx by i238212
1st parposal presentation.pptx1st parposal presentation.pptx
1st parposal presentation.pptx
i2382129 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson66 views
HTTP headers that make your website go faster - devs.gent November 2023 by Thijs Feryn
HTTP headers that make your website go faster - devs.gent November 2023HTTP headers that make your website go faster - devs.gent November 2023
HTTP headers that make your website go faster - devs.gent November 2023
Thijs Feryn21 views
Transcript: The Details of Description Techniques tips and tangents on altern... by BookNet Canada
Transcript: The Details of Description Techniques tips and tangents on altern...Transcript: The Details of Description Techniques tips and tangents on altern...
Transcript: The Details of Description Techniques tips and tangents on altern...
BookNet Canada135 views

Scaling an Extreme Temporary Event Network for Burning Man

  • 1. Burning Man Scaling an extreme temporary event network Matt Peterson matt@burningman.com Matt Peterson
  • 2. Terry Ratcliff Reuters/Jim Urquhart Black Rock City, NV duncan.co
  • 3. 2 01 4 B ur n i ng M an IP N etwo r k Team I.T.S. is Backbone, Camera Girl, Cat, Domo, Huckleberry, Little Meat, MattStep, Mushroom, Prof. Fox, PornStar, Ralf, Reset, Sawdust, Spank Me, Taz, Whiskey Devil, and Wild Card gw-noc lo0 - 162.212.145.252 Primary Cost 100 Tertiary Cost 400 Secondary Cost 150 Channel 5 TX 11.385GHz RX 10.895GHz gw-depot lo0 - 162.212.145.254 Channel 2 TX 11.265GHz RX 10.775GHz Secondary Cost 150 Channel 150 5.750GHz Channel 140 TX 5.700GHz RX 5.750GHz dw-tower-noc port1 - 162.212.144.26 dw-noc-tower port1 - 162.212.144.27 High Desert Internet dw-tower-depot port1 - 162.212.144.10 dw-depot-tower port1 - 162.212.144.11 dw-noc-depot port1 - 162.212.144.75 dw-depot-noc port1 - 162.212.144.74 ap-tower br0.2 - 162.212.144.2 st-noc-tower br0.6 - 162.212.144.34 Poito Eagle Ridge st-depot-tower br0.4 - 162.212.144.18 ge0/3 - 162.212.144.25 ge0/2 - 162.212.144.9 Primary Cost 100 Tertiary Cost 400 m a t t @ b u r n i n g m a n . c o m 2 0 1 4 - 1 0 - 0 2 1 8 : 1 0 Channel 5 TX 10.895GHz RX 11.385GHz Channel 2 TX 10.775GHz RX 11.265GHz Channel 1 TX 17.765GHz RX 19.325GHz Channel 1 TX 19.325GHz RX 17.765GHz sw-noc2 vl7 - 162.212.144.123 sw-noc1 vl7 - 162.212.144.122 v94 - NOC LAN v111 - First Camp v114 - Media Mecca v115 - NOC Inside v121 - Webcast v94 - NOC LAN v103 - Artery v210 - Digerati & Devas ap-noc2 br0.25 - 162.212.144.146 ap-noc10 br0.25 - 162.212.144.149 ap-noc7 br0.25 - 162.212.144.148 sw-depot vl8 - 162.212.144.130 Channel 6 2.437GHz apb-noc-local br0 - 100.96.94.2 Channel 1 2.412GHz apb-depot-local br0 - 100.96.91.2 Channel 1 2.412GHz apb-noc1-inside br0 - 100.96.115.2 v122 - OC1/West Wing Radio Frequency UTP Ethernet Licensed Unlicensed OSPF weight Transit Channel 147 5.735GHz Channel 153 5.765GHz Channel 151 5.755GHz Channel 157 5.785GHz apa-noc br0.33 - 162.212.145.35 Channel 165 5.825GHz v91 - Depot, Logistics st-light-depot br0.17 - 162.212.144.210 st-light-noc br0.27 - 162.212.144.202 st-airport br0.101 - 100.96.101.2 v107 - Commissary Office v113 - Laminates v125 - Big Office v126 - Human Resources v127 - Container Office gw-light lo0 - 162.212.145.248 v100 - Accounting v197 - Commissary Public v215 - GPE Camp v221 - Ticketfly v224 - Boob sw-commissary mgmt.34 - 162.212.146.142 ap-depot-omni br0.10 - 162.212.144.66 Channel 163 5.815GHz Channel 11 2.462GHz apb-commissary-local br0 - 100.96.107.2 st-poopdudes br0.216 - 100.96.216.2 st-quad4-depot br0.15 - 162.212.144.106 st-quad4-noc br0.23 - 162.212.144.170 v104 - BMIR v108 - DMV v120 - Ranger HQ gw-quad4 lo0 - 162.212.145.249 v118 - Playa Info v120 - Ranger HQ sw-quad4 mgmt.35 - 162.212.146.130 Channel 6 2.437GHz apb-quad4-local br0 - 100.96.95.2 st-cafe br0.216 - 100.96.216.2 Channel 11 2.462GHz apb-cafe-local br0 - 100.96.106.3 st-pgepoint1 br0.117 - 100.96.117.2 Channel 6 2.437GHz apb-pgepoint1 br0 - 100.96.117.4 st-ghetto br0.211 - 100.96.211.2 Channel 1 2.412GHz apb-ghetto-local br0 - 100.96.211.3 br0.22 - 162.212.144.162 gw-esd st-esd-depot br0.14 - 162.212.144.98 st-esd-noc lo0 - 162.212.145.253 v20 - ESD Comm. v111 - Incident Comm. Post st-box-depot br0.13 - 162.212.144.90 st-box-noc br0.21 - 162.212.144.154 gw-box lo0 - 162.212.145.250 v105 - Box Office v116 - GPE st-power-depot br0.18 - 162.212.144.218 st-power-noc br0.28 - 162.212.144.226 gw-power lo0 - 162.212.145.247 v119 - Power Camp Channel 11 2.462GHz apb-power-local br0 - 100.96.97.2 st-heavy-depot br0.16 - 162.212.144.114 st-heavy-noc br0.24 - 162.212.144.178 gw-heavy lo0 - 162.212.145.251 v96 - Heavy Camp v109 - HGH "Rampart" v112 - Heavy Office Channel 6 2.437GHz apb-heavy-local br0 - 100.96.96.2 st-esdstation9 br0.130 - 100.96.130.2 st-esdstation3 br0.129 - 100.96.129.2 ap-noc5 br0.25 - 162.212.144.147 ap-depot-sector br0.10 - 162.212.144.67 Channel 153 5.795GHz Channel 6 2.437GHz apb-depot-dispatch br0 - 100.96.91.3
  • 4. Where We Started • The network worked, but it wasn’t easy – Large L2 bridged architecture, minimal L3 segmentation, multiple NAT layers – Two distinct “business units” – Manual configuration, “tribal knowledge” – Numerous single points of failure
  • 5. Where We Went • Needed to operate as a unified team – Consistent support experience, improved RF spectral efficiency, coordinated IP allocations • Standardized COTS equipment – “CCIE off the street” factor, escalation path • Standardized service offerings – Org department handoff’s always wired gigE; as aggregated “islands” or single demarc – Participant camps supply very prescriptive equipment, “self-install” provisioning
  • 6. Where We Went • Route, always – No L2 segments past a single device – OSPF everywhere, core backbone & “islands” – Segment where possible, even over WiFi • Automation – Initially covering all routers & switches – Target goal to cover any device with a config or supplemental service (DNS, monitoring)
  • 7. Automation! • Held bakeoff (mid 2011 evaluation) – Homegrown YAML  config templates – Prototyped NCG (see NANOG49 Tutorial “Automating Network Configuration”) • NCG won (3 yrs ago) – Open source, vendor agnostic – Initial steep curve, very easy to embrace – Principal developer already a team member
  • 10. Summary Overview • {Automation} data modeling isn’t easy – Imagine all your inputs & outputs (device configs, DNS, monitoring, billing, etc.) • Single source of truth – Git, a wiki, fancy IPAM: choose what fits your organization’s workflow, stress level, & budget • Start at L8 and L1, meet in the middle – People + physical layer = organic processes – End goal is to be efficient, not become a SW dev
  • 11. More.. unconference? Matt Peterson matt@burningman.com

Editor's Notes

  1. Thank you for the opportunity this morning Originally just automation - background on design, technology
  2. Burning Man is an event … held on federal land … northern Nevada, 2hrs north-east of Reno Called the playa or Black Rock City Just under 70k participants Described as social experiment, festival, party, … - it’s a city Leave no trace event Zero infrastructure before & after event ~3 weeks to build out & 3 weeks to tear down BM provides basic life safety (port-a-potties, medical care) along with “guard rails” (mutual aid – law enforcement, ice for purchase) Everything else – water, food, shelter, Bring Your Own
  3. Everyone city has infrastructure This is what BM looks like to me
  4. When I returned, took the approach of Toyota manufacturing, Six Sigma, consultant evaluation phase Team = sysadmin/helpdesk, not network engineers or architects Some level of routing, done with shell scripts that adjusted local routing tables Limited investment to add redundancy, power outages and physical stability For historical reasons, two different customer bases Departments pre-event, camps during = staff exhausted
  5. Common team with common goal of an effective service, regardless if the end-user is a department OR camp COTS is an old term, “commercial off the shelf” – products that offer technical support, warranty, known best practices For switching - wanted fanless, active PoE, gigE Handoff isn’t known to the customer, be it wired or wireless backhaul A truck roll is incredibly expensive
  6. Same L2 across playa, bring your own VPN or tunnel mechanism OSPF just works, most devices L2 bridges – no need for exotic mesh networking Tired of sitting in a shipping container for a week, too much hands-on L1 work to be done Can’t afford to manually config
  7. Consistent service, standardized equipment = made it easier to automate! You don’t have to buy all the same equipment, but in our circumstances, it helped
  8. Only did switch + router configs, then added DNS, monitoring Added wireless equipment later Take baby steps NCG + static configs in Git very powerful, offline distributed database Change is difficult for everyone to handle, tackle people and physical layer first = automation becomes a natural extension