1.
How to run a global, cloud scale event for
10.000 people

2.
How to run a global, cloud scale event for
10.000 people
@ROBBOS81

3.
35 countries
10.000 participants
1340 web apps
4500 YouTube views
1530 resource groups 7 Azure DevOps organizations
4 Azure subscriptions
3 outages in Azure DevOps
Half of the budget
Dedicated Microsoft SRE’s on call
36 hours
4 million impressions
Free community event

4.
WHAT’S IN IT FOR YOU?
This does not only work for community events
Shows working together in a globally distributed team
Technical design decisions
Design with the end in mind

5.
@ROBBOS81
WHAT IS GDBC?
• Global DevOps BootCamp
• Free
• Community event
• Saturday of learning

6.
@ROBBOS81
ORIGINATION
• DevOps as a topic
• Issues with other bootcamps:
• Only global by name
• Create your own:
• content
• workshop material
• Lots of work
EVENT OUT OF THE
BOX

7.
@ROBBOS81
EVENT OUT OF THE BOX
Worldwide event
Same content
Same exercises
Global vibe

8.
@ROBBOS81
EVENT OUT OF THE BOX
FOR LOCAL ORGANIZERS
Provide location
WIFI
Host the local event
Local speaker
Provide enough proctors
Engage local community

9.
• Global keynote
2018: Buck Hodges
Director of Engineering,
Azure DevOps
2019: Niall Murphy
Global Head of Azure SRE
2017: Donovan Brown
Cloud Advocate Manager,
Methods and Practices
EVENT OUT OF THE BOX
FOR US

10.
@ROBBOS81
EVENT OUT OF THE BOX
FOR US
• Global keynote
• Local keynote
• Content around a theme
• Exercises for attendees
• Scoreboard
• Infrastructure
• Communication
• Marketing & Branding
• Social media

11.
GDBC CORE TEAM
Team to create + run the global event
Volunteers from the community
Planning / meetings
Sponsoring

12.
SEARCH FOR VENUE ORGANIZERS
Get local venues:
• Google Forms + website
• MVP Summit
• Radio TFS
Marketing:
• Self-promotion
• Local communities

13.
IS KEY
Build a community of local venues and GDBC Core Team
• Slack
• Community calls
• Record and reshare
• Centralized wiki

14.
STAKEHOLDER MANAGEMENT

15.
@ROBBOS81
WHO ARE WE DOING IT FOR?
90 venues worldwide

16.
WHAT DO THE VENUES NEED?
Attendees
Content

17.
@ROBBOS81
ATTENDEES
Platform to handle:
• Attendee registration
• Local landing page
• Social marketing
Eventbrite
• Platform as a service
• Venue organizers

18.
ATTENDEES
Eventbrite
• API is not complete
• Co-admins are hard
• Not everything can be automated
Automate as much as possible
• ConsoleApp with Selenium to add co-admins
• Auto-invite to Slack

19.
SAAS OVER PAAS OVER IAAS

20.
REACHING ATTENDEES - SOCIAL

21.
PREPARING THE CONTENT
Keynotes
Exercises
Workspace
Video content
Styling Guide
Theme

22.
THEME 2018 THEME 2019THEME 2017

23.
WORKSPACE

24.
CHALLENGES
•SSL certificate expired

25.
CHALLENGES
•SSL certificate expired
•Flaky connection
•Credential leak
•Exception rate goes up
•DDOS after CEO message
•Supply chain attack

26.
DDOS AFTER CEO MESSAGE

27.
@ROBBOS81
COMMUNITY
• What is GDBC
• Webshop
• Challenges

28.
SETTING UP AZURE

29.
@ROBBOS81
WORKING WEBSHOP
+
+

30.
@ROBBOS81
WHAT DO WE NEED
Resource group
App Service
SQL Server Database
Application Insights
Azure Active Directory
• Venue admin account
• Per team:
• User account + user group
• Service principal for Azure DevOps
X 1200

31.
REGIONALLY DIVIDED
•Spread the load
•Reduce latency for users

32.
@ROBBOS81
USE WHAT IS BEST FOR YOU

33.
AZURE DEVOPS TO THE RESCUE

34.
• SQL Servers India: max capacity reached
• Application Insights Central US unavailable
• Resource types off by default

35.
SQL SERVER LIMITS
20 per region per subscription
200 max. per subscription

36.
2000
role assignments
per subscription

37.
DON’T TRUST THE DEFAULT
• During testing, creating a default SQL was a S1 Database
• Cost: € 25 / month
• During the last week, the default change to a DS3v1
• Cost: € 315 / month
• We created 1200 databases…..

38.
COST MANAGEMENT

39.
AZURE HAS BEEN TACKLED

40.
AZURE DEVOPS

41.
@ROBBOS81
REQUIREMENTS FOR AZURE DEVOPS
Organization
Team project
Git repository
Build pipeline (CI)
Deployment pipeline (CD)
Service connection to Azure
Artifact feed
Azure Active Directory Link
X 1200

42.
@ROBBOS81
AZURE DEVOPS PROVISIONING
Multiple organizations to spread the load:
Use a service account for setup
• Australia
• Brazil
• Canada
• East Asia
• West Europe
• India
• United Kingdom
• United States

43.
BUILD AGENTS
7
organizations
1200
teams
±200
teams per org
1-10
concurrent
pipelines
150
sponsored
pipelines
per org:

44.
@ROBBOS81

45.
@ROBBOS81
Every day in preparation
Azure Infrastructure
Azure DevOps
Certificates

46.
@ROBBOS81

47.
LEARNINGS AZURE DEVOPS PROVISIONING
Hitting the service at scale can trigger some weird issues
• Build pipeline outage
• Two regions
• Quick fix: 1 concurrent pipeline
• Australia networking outage
• Agent scale set network issue

48.
@ROBBOS81
AZURE DEVOPS – PRODUCT TEAM
Responsive team
24 hours 3 SRE’s assigned

49.
SPONSOR BUY IN

50.
AZURE DEVOPS

51.
@ROBBOS81
EVENT DAY

52.
@ROBBOS81
CHALLENGES WEBSITE
1. Explanation
2. Detect
3. Respond: Quick fix
4. Post-mortem
5. Recover

53.
@ROBBOS81
CHALLENGE
Docker containers to disrupt the webshop
Start, stop, validate and scoring
• Isolated
• Own technology stack
• Parameters injected

54.
@ROBBOS81
CONTAINER REQUIREMENTS
• Asynchronous
• Fast
• Scalable
X 30.000

55.
@ROBBOS81
WHERE TO RUN
Azure Container Instances
No initial setup
Slow start of containers
Soft limits
Azure Kubernetes Services
Provision cluster
Limited to available hardware
Limited to nodes, scale up

56.
@ROBBOS81
MONITORING

57.
@ROBBOS81
AKS CLUSTER MONITORING

58.
YOU BUILD IT, YOU RUN IT
36
hours

59.
RISK MANAGEMENT

60.
@ROBBOS81
LEARNINGS
36 hours monitoring is hard!
Find an SRE in a different time zone
• Preparation is key
• Isolation and independency
• Caching and scaling really helps
• Insights and control
4:00 AM

61.
CUSTOMER HAPPINESS

62.
TAKE AWAYS
START WITH A VISION
DESIGN WITH THE END IN MIND
ENGAGE SPONSORS AND
STAKEHOLDERS
THINK BIG
BUILD VS BUY
RISK MANAGEMENT
CUSTOMER HAPPINESS

63.
@ROBBOS81
MOST IMPORTANTLY
DON’T THINK,
ACT!

64.
@ROBBOS81
LOCAL DEVOPS BOOTCAMP
https://localdevopsbootcamp.com
• SSL certificate expired
• Flaky connection
• Credential leak
• Exception rate goes up
• DDOS after CEO message
• Supply chain attack

65.
https://xpir.it/LinksGDBC
ROB BOS - @ROBBOS81

66.
How to run a global, cloud scale event for 10.000
people
@ROBBOS81