SlideShare a Scribd company logo
How to run a global, cloud scale event for
10.000 people
How to run a global, cloud scale event for
10.000 people
@ROBBOS81
35 countries
10.000 participants
1340 web apps
4500 YouTube views
1530 resource groups 7 Azure DevOps organizations
4 Azure subscriptions
3 outages in Azure DevOps
Half of the budget
Dedicated Microsoft SRE’s on call
36 hours
4 million impressions
Free community event
WHAT’S IN IT FOR YOU?
This does not only work for community events
Shows working together in a globally distributed team
Technical design decisions
Design with the end in mind
@ROBBOS81
WHAT IS GDBC?
• Global DevOps BootCamp
• Free
• Community event
• Saturday of learning
@ROBBOS81
ORIGINATION
• DevOps as a topic
• Issues with other bootcamps:
• Only global by name
• Create your own:
• content
• workshop material
• Lots of work
EVENT OUT OF THE
BOX
@ROBBOS81
EVENT OUT OF THE BOX
Worldwide event
Same content
Same exercises
Global vibe
@ROBBOS81
EVENT OUT OF THE BOX
FOR LOCAL ORGANIZERS
Provide location
WIFI
Host the local event
Local speaker
Provide enough proctors
Engage local community
• Global keynote
2018: Buck Hodges
Director of Engineering,
Azure DevOps
2019: Niall Murphy
Global Head of Azure SRE
2017: Donovan Brown
Cloud Advocate Manager,
Methods and Practices
EVENT OUT OF THE BOX
FOR US
@ROBBOS81
EVENT OUT OF THE BOX
FOR US
• Global keynote
• Local keynote
• Content around a theme
• Exercises for attendees
• Scoreboard
• Infrastructure
• Communication
• Marketing & Branding
• Social media
GDBC CORE TEAM
Team to create + run the global event
Volunteers from the community
Planning / meetings
Sponsoring
SEARCH FOR VENUE ORGANIZERS
Get local venues:
• Google Forms + website
• MVP Summit
• Radio TFS
Marketing:
• Self-promotion
• Local communities
IS KEY
Build a community of local venues and GDBC Core Team
• Slack
• Community calls
• Record and reshare
• Centralized wiki
STAKEHOLDER MANAGEMENT
@ROBBOS81
WHO ARE WE DOING IT FOR?
90 venues worldwide
WHAT DO THE VENUES NEED?
Attendees
Content
@ROBBOS81
ATTENDEES
Platform to handle:
• Attendee registration
• Local landing page
• Social marketing
Eventbrite
• Platform as a service
• Venue organizers
ATTENDEES
Eventbrite
• API is not complete
• Co-admins are hard
• Not everything can be automated
Automate as much as possible
• ConsoleApp with Selenium to add co-admins
• Auto-invite to Slack
SAAS OVER PAAS OVER IAAS
REACHING ATTENDEES - SOCIAL
PREPARING THE CONTENT
Keynotes
Exercises
Workspace
Video content
Styling Guide
Theme
THEME 2018 THEME 2019THEME 2017
WORKSPACE
CHALLENGES
•SSL certificate expired
CHALLENGES
•SSL certificate expired
•Flaky connection
•Credential leak
•Exception rate goes up
•DDOS after CEO message
•Supply chain attack
DDOS AFTER CEO MESSAGE
@ROBBOS81
COMMUNITY
• What is GDBC
• Webshop
• Challenges
SETTING UP AZURE
@ROBBOS81
WORKING WEBSHOP
+
+
@ROBBOS81
WHAT DO WE NEED
Resource group
App Service
SQL Server Database
Application Insights
Azure Active Directory
• Venue admin account
• Per team:
• User account + user group
• Service principal for Azure DevOps
X 1200
REGIONALLY DIVIDED
•Spread the load
•Reduce latency for users
@ROBBOS81
USE WHAT IS BEST FOR YOU
AZURE DEVOPS TO THE RESCUE
• SQL Servers India: max capacity reached
• Application Insights Central US unavailable
• Resource types off by default
SQL SERVER LIMITS
20 per region per subscription
200 max. per subscription
2000
role assignments
per subscription
DON’T TRUST THE DEFAULT
• During testing, creating a default SQL was a S1 Database
• Cost: € 25 / month
• During the last week, the default change to a DS3v1
• Cost: € 315 / month
• We created 1200 databases…..
COST MANAGEMENT
AZURE HAS BEEN TACKLED
AZURE DEVOPS
@ROBBOS81
REQUIREMENTS FOR AZURE DEVOPS
Organization
Team project
Git repository
Build pipeline (CI)
Deployment pipeline (CD)
Service connection to Azure
Artifact feed
Azure Active Directory Link
X 1200
@ROBBOS81
AZURE DEVOPS PROVISIONING
Multiple organizations to spread the load:
Use a service account for setup
• Australia
• Brazil
• Canada
• East Asia
• West Europe
• India
• United Kingdom
• United States
BUILD AGENTS
7
organizations
1200
teams
±200
teams per org
1-10
concurrent
pipelines
150
sponsored
pipelines
per org:
@ROBBOS81
@ROBBOS81
Every day in preparation
Azure Infrastructure
Azure DevOps
Certificates
@ROBBOS81
LEARNINGS AZURE DEVOPS PROVISIONING
Hitting the service at scale can trigger some weird issues
• Build pipeline outage
• Two regions
• Quick fix: 1 concurrent pipeline
• Australia networking outage
• Agent scale set network issue
@ROBBOS81
AZURE DEVOPS – PRODUCT TEAM
Responsive team
24 hours 3 SRE’s assigned
SPONSOR BUY IN
AZURE DEVOPS
@ROBBOS81
EVENT DAY
@ROBBOS81
CHALLENGES WEBSITE
1. Explanation
2. Detect
3. Respond: Quick fix
4. Post-mortem
5. Recover
@ROBBOS81
CHALLENGE
Docker containers to disrupt the webshop
Start, stop, validate and scoring
• Isolated
• Own technology stack
• Parameters injected
@ROBBOS81
CONTAINER REQUIREMENTS
• Asynchronous
• Fast
• Scalable
X 30.000
@ROBBOS81
WHERE TO RUN
Azure Container Instances
No initial setup
Slow start of containers
Soft limits
Azure Kubernetes Services
Provision cluster
Limited to available hardware
Limited to nodes, scale up
@ROBBOS81
MONITORING
@ROBBOS81
AKS CLUSTER MONITORING
YOU BUILD IT, YOU RUN IT
36
hours
RISK MANAGEMENT
@ROBBOS81
LEARNINGS
36 hours monitoring is hard!
Find an SRE in a different time zone
• Preparation is key
• Isolation and independency
• Caching and scaling really helps
• Insights and control
4:00 AM
CUSTOMER HAPPINESS
TAKE AWAYS
START WITH A VISION
DESIGN WITH THE END IN MIND
ENGAGE SPONSORS AND
STAKEHOLDERS
THINK BIG
BUILD VS BUY
RISK MANAGEMENT
CUSTOMER HAPPINESS
@ROBBOS81
MOST IMPORTANTLY
DON’T THINK,
ACT!
@ROBBOS81
LOCAL DEVOPS BOOTCAMP
https://localdevopsbootcamp.com
• SSL certificate expired
• Flaky connection
• Credential leak
• Exception rate goes up
• DDOS after CEO message
• Supply chain attack
https://xpir.it/LinksGDBC
ROB BOS - @ROBBOS81
How to run a global, cloud scale event for 10.000
people
@ROBBOS81

More Related Content

What's hot

Standardizing and Managing Your Infrastructure - MOSC 2011
Standardizing and Managing Your Infrastructure - MOSC 2011Standardizing and Managing Your Infrastructure - MOSC 2011
Standardizing and Managing Your Infrastructure - MOSC 2011
Brian Ritchie
 
Accelerating DevOps Pipelines with AWS
Accelerating DevOps Pipelines with AWSAccelerating DevOps Pipelines with AWS
Accelerating DevOps Pipelines with AWS
Amazon Web Services
 
Why BaaS is crucial to early stage startups
Why BaaS is crucial to early stage startupsWhy BaaS is crucial to early stage startups
Why BaaS is crucial to early stage startups
Jane Chung
 
Cloud Native Camel Riding
Cloud Native Camel RidingCloud Native Camel Riding
Cloud Native Camel Riding
Christian Posta
 
Best practices deploying Sitecore to Microsoft Azure
Best practices deploying Sitecore to Microsoft AzureBest practices deploying Sitecore to Microsoft Azure
Best practices deploying Sitecore to Microsoft Azure
Thom Puiman
 
Event driven infrastructure
Event driven infrastructureEvent driven infrastructure
Event driven infrastructure
Shiva Narayanaswamy
 
Alfresco Digital Business Platform - Why, How, What
Alfresco Digital Business Platform - Why, How, WhatAlfresco Digital Business Platform - Why, How, What
Alfresco Digital Business Platform - Why, How, What
Mario Romano
 
Bitbucket Pipelines: Serverless CI/CD That Will Save Your Life
Bitbucket Pipelines: Serverless CI/CD That Will Save Your LifeBitbucket Pipelines: Serverless CI/CD That Will Save Your Life
Bitbucket Pipelines: Serverless CI/CD That Will Save Your Life
Atlassian
 
DevCon 2018 - 5 ways to use AWS with Alfresco
DevCon 2018 - 5 ways to use AWS with AlfrescoDevCon 2018 - 5 ways to use AWS with Alfresco
DevCon 2018 - 5 ways to use AWS with Alfresco
Gavin Cornwell
 
Get your head in the clouds! - Swansea Con 2016
Get your head in the clouds! - Swansea Con 2016Get your head in the clouds! - Swansea Con 2016
Get your head in the clouds! - Swansea Con 2016
Christopher Cundill
 
Serverless On Stage - Serverless URL Shortener
Serverless On Stage - Serverless URL ShortenerServerless On Stage - Serverless URL Shortener
Serverless On Stage - Serverless URL Shortener
Luca Bianchi
 
CI/CD and Asset Serving for Single Page Apps
CI/CD and Asset Serving for Single Page AppsCI/CD and Asset Serving for Single Page Apps
CI/CD and Asset Serving for Single Page Apps
Mike North
 
Serverless Meetup - 12 gennaio 2017
Serverless Meetup - 12 gennaio 2017Serverless Meetup - 12 gennaio 2017
Serverless Meetup - 12 gennaio 2017
Luca Bianchi
 
AWS DevOps Event - Innovating with DevOps on AWS
AWS DevOps Event - Innovating with DevOps on AWSAWS DevOps Event - Innovating with DevOps on AWS
AWS DevOps Event - Innovating with DevOps on AWS
Ian Massingham
 
DevOps and AWS
DevOps and AWSDevOps and AWS
DevOps and AWS
Shiva Narayanaswamy
 
Alfresco Process Services REST API - Alfresco DevCon 2018
 Alfresco Process Services REST API - Alfresco DevCon 2018 Alfresco Process Services REST API - Alfresco DevCon 2018
Alfresco Process Services REST API - Alfresco DevCon 2018
Dennis Koch
 
DevOps from a developer perspective
DevOps from a developer perspective DevOps from a developer perspective
DevOps from a developer perspective
WSO2
 
Zero Downtime Deployment
Zero Downtime DeploymentZero Downtime Deployment
Zero Downtime Deployment
Joel Dickson
 
Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...
Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...
Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...
Edward Wilde
 
How to Grow a Serverless Team
How to Grow a Serverless TeamHow to Grow a Serverless Team
How to Grow a Serverless Team
SheenBrisals
 

What's hot (20)

Standardizing and Managing Your Infrastructure - MOSC 2011
Standardizing and Managing Your Infrastructure - MOSC 2011Standardizing and Managing Your Infrastructure - MOSC 2011
Standardizing and Managing Your Infrastructure - MOSC 2011
 
Accelerating DevOps Pipelines with AWS
Accelerating DevOps Pipelines with AWSAccelerating DevOps Pipelines with AWS
Accelerating DevOps Pipelines with AWS
 
Why BaaS is crucial to early stage startups
Why BaaS is crucial to early stage startupsWhy BaaS is crucial to early stage startups
Why BaaS is crucial to early stage startups
 
Cloud Native Camel Riding
Cloud Native Camel RidingCloud Native Camel Riding
Cloud Native Camel Riding
 
Best practices deploying Sitecore to Microsoft Azure
Best practices deploying Sitecore to Microsoft AzureBest practices deploying Sitecore to Microsoft Azure
Best practices deploying Sitecore to Microsoft Azure
 
Event driven infrastructure
Event driven infrastructureEvent driven infrastructure
Event driven infrastructure
 
Alfresco Digital Business Platform - Why, How, What
Alfresco Digital Business Platform - Why, How, WhatAlfresco Digital Business Platform - Why, How, What
Alfresco Digital Business Platform - Why, How, What
 
Bitbucket Pipelines: Serverless CI/CD That Will Save Your Life
Bitbucket Pipelines: Serverless CI/CD That Will Save Your LifeBitbucket Pipelines: Serverless CI/CD That Will Save Your Life
Bitbucket Pipelines: Serverless CI/CD That Will Save Your Life
 
DevCon 2018 - 5 ways to use AWS with Alfresco
DevCon 2018 - 5 ways to use AWS with AlfrescoDevCon 2018 - 5 ways to use AWS with Alfresco
DevCon 2018 - 5 ways to use AWS with Alfresco
 
Get your head in the clouds! - Swansea Con 2016
Get your head in the clouds! - Swansea Con 2016Get your head in the clouds! - Swansea Con 2016
Get your head in the clouds! - Swansea Con 2016
 
Serverless On Stage - Serverless URL Shortener
Serverless On Stage - Serverless URL ShortenerServerless On Stage - Serverless URL Shortener
Serverless On Stage - Serverless URL Shortener
 
CI/CD and Asset Serving for Single Page Apps
CI/CD and Asset Serving for Single Page AppsCI/CD and Asset Serving for Single Page Apps
CI/CD and Asset Serving for Single Page Apps
 
Serverless Meetup - 12 gennaio 2017
Serverless Meetup - 12 gennaio 2017Serverless Meetup - 12 gennaio 2017
Serverless Meetup - 12 gennaio 2017
 
AWS DevOps Event - Innovating with DevOps on AWS
AWS DevOps Event - Innovating with DevOps on AWSAWS DevOps Event - Innovating with DevOps on AWS
AWS DevOps Event - Innovating with DevOps on AWS
 
DevOps and AWS
DevOps and AWSDevOps and AWS
DevOps and AWS
 
Alfresco Process Services REST API - Alfresco DevCon 2018
 Alfresco Process Services REST API - Alfresco DevCon 2018 Alfresco Process Services REST API - Alfresco DevCon 2018
Alfresco Process Services REST API - Alfresco DevCon 2018
 
DevOps from a developer perspective
DevOps from a developer perspective DevOps from a developer perspective
DevOps from a developer perspective
 
Zero Downtime Deployment
Zero Downtime DeploymentZero Downtime Deployment
Zero Downtime Deployment
 
Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...
Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...
Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...
 
How to Grow a Serverless Team
How to Grow a Serverless TeamHow to Grow a Serverless Team
How to Grow a Serverless Team
 

Similar to How to run a global, cloud scale event for 10.000 people

Practical PaaS presentation
Practical PaaS presentationPractical PaaS presentation
Practical PaaS presentation
hmalphettes
 
Docker Hub Breakout Session at DockerCon by Ken Cochrane
Docker Hub Breakout Session at DockerCon by Ken CochraneDocker Hub Breakout Session at DockerCon by Ken Cochrane
Docker Hub Breakout Session at DockerCon by Ken Cochrane
Docker, Inc.
 
My personal story from azure it pro to azure dev ops
My personal story from azure it pro to azure dev opsMy personal story from azure it pro to azure dev ops
My personal story from azure it pro to azure dev ops
nj-azure
 
AWS re:Invent 2016: AWS Training Opportunities (DCS202 )
AWS re:Invent 2016: AWS Training Opportunities (DCS202 )AWS re:Invent 2016: AWS Training Opportunities (DCS202 )
AWS re:Invent 2016: AWS Training Opportunities (DCS202 )
Amazon Web Services
 
Untangling the web11
Untangling the web11Untangling the web11
Untangling the web11
Derek Jacoby
 
Neo4j for Cloud Management at Scale
Neo4j for Cloud Management at ScaleNeo4j for Cloud Management at Scale
Neo4j for Cloud Management at Scale
Neo4j
 
AWS Kochi User Group Presentation
AWS  Kochi User Group PresentationAWS  Kochi User Group Presentation
AWS Kochi User Group Presentation
Varun Manik
 
Why to Cloud Native
Why to Cloud NativeWhy to Cloud Native
Why to Cloud Native
Karthik Gaekwad
 
EWUG 2018 February the journey continues.....
EWUG 2018 February   the journey continues.....EWUG 2018 February   the journey continues.....
EWUG 2018 February the journey continues.....
Peter Selch Dahl
 
Micro Services in .NET Core and Docker
Micro Services in .NET Core and DockerMicro Services in .NET Core and Docker
Micro Services in .NET Core and Docker
cjmyers
 
Lessons learned on the Azure API Stewardship Journey.pptx
Lessons learned on the Azure API Stewardship Journey.pptxLessons learned on the Azure API Stewardship Journey.pptx
Lessons learned on the Azure API Stewardship Journey.pptx
apidays
 
(APP303) Lightning Fast Deploys with Docker Containers and AWS | AWS re:Inven...
(APP303) Lightning Fast Deploys with Docker Containers and AWS | AWS re:Inven...(APP303) Lightning Fast Deploys with Docker Containers and AWS | AWS re:Inven...
(APP303) Lightning Fast Deploys with Docker Containers and AWS | AWS re:Inven...
Amazon Web Services
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Adrian Cockcroft
 
Development & Test on AWS - IP Expo 2013
Development & Test on AWS - IP Expo 2013Development & Test on AWS - IP Expo 2013
Development & Test on AWS - IP Expo 2013
Amazon Web Services
 
Lean Startup with WebObjects
Lean Startup with WebObjectsLean Startup with WebObjects
Lean Startup with WebObjects
WO Community
 
Aws amity university_presentation_oct-31-2020
Aws amity university_presentation_oct-31-2020Aws amity university_presentation_oct-31-2020
Aws amity university_presentation_oct-31-2020
Varun Manik
 
Webcast: DevOps in AWS is different! How can containers help?
Webcast: DevOps in AWS is different! How can containers help? Webcast: DevOps in AWS is different! How can containers help?
Webcast: DevOps in AWS is different! How can containers help?
Applatix
 
DevOps Columbus Meetup Kickoff - Infrastructure as Code
DevOps Columbus Meetup Kickoff - Infrastructure as CodeDevOps Columbus Meetup Kickoff - Infrastructure as Code
DevOps Columbus Meetup Kickoff - Infrastructure as Code
Michael Ducy
 
Containers, microservices and serverless for realists
Containers, microservices and serverless for realistsContainers, microservices and serverless for realists
Containers, microservices and serverless for realists
Karthik Gaekwad
 
"Portrait of the developer as The Artist" Lockheed Architect Workshop
"Portrait of the developer as The Artist" Lockheed Architect Workshop"Portrait of the developer as The Artist" Lockheed Architect Workshop
"Portrait of the developer as The Artist" Lockheed Architect Workshop
Patrick Chanezon
 

Similar to How to run a global, cloud scale event for 10.000 people (20)

Practical PaaS presentation
Practical PaaS presentationPractical PaaS presentation
Practical PaaS presentation
 
Docker Hub Breakout Session at DockerCon by Ken Cochrane
Docker Hub Breakout Session at DockerCon by Ken CochraneDocker Hub Breakout Session at DockerCon by Ken Cochrane
Docker Hub Breakout Session at DockerCon by Ken Cochrane
 
My personal story from azure it pro to azure dev ops
My personal story from azure it pro to azure dev opsMy personal story from azure it pro to azure dev ops
My personal story from azure it pro to azure dev ops
 
AWS re:Invent 2016: AWS Training Opportunities (DCS202 )
AWS re:Invent 2016: AWS Training Opportunities (DCS202 )AWS re:Invent 2016: AWS Training Opportunities (DCS202 )
AWS re:Invent 2016: AWS Training Opportunities (DCS202 )
 
Untangling the web11
Untangling the web11Untangling the web11
Untangling the web11
 
Neo4j for Cloud Management at Scale
Neo4j for Cloud Management at ScaleNeo4j for Cloud Management at Scale
Neo4j for Cloud Management at Scale
 
AWS Kochi User Group Presentation
AWS  Kochi User Group PresentationAWS  Kochi User Group Presentation
AWS Kochi User Group Presentation
 
Why to Cloud Native
Why to Cloud NativeWhy to Cloud Native
Why to Cloud Native
 
EWUG 2018 February the journey continues.....
EWUG 2018 February   the journey continues.....EWUG 2018 February   the journey continues.....
EWUG 2018 February the journey continues.....
 
Micro Services in .NET Core and Docker
Micro Services in .NET Core and DockerMicro Services in .NET Core and Docker
Micro Services in .NET Core and Docker
 
Lessons learned on the Azure API Stewardship Journey.pptx
Lessons learned on the Azure API Stewardship Journey.pptxLessons learned on the Azure API Stewardship Journey.pptx
Lessons learned on the Azure API Stewardship Journey.pptx
 
(APP303) Lightning Fast Deploys with Docker Containers and AWS | AWS re:Inven...
(APP303) Lightning Fast Deploys with Docker Containers and AWS | AWS re:Inven...(APP303) Lightning Fast Deploys with Docker Containers and AWS | AWS re:Inven...
(APP303) Lightning Fast Deploys with Docker Containers and AWS | AWS re:Inven...
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
 
Development & Test on AWS - IP Expo 2013
Development & Test on AWS - IP Expo 2013Development & Test on AWS - IP Expo 2013
Development & Test on AWS - IP Expo 2013
 
Lean Startup with WebObjects
Lean Startup with WebObjectsLean Startup with WebObjects
Lean Startup with WebObjects
 
Aws amity university_presentation_oct-31-2020
Aws amity university_presentation_oct-31-2020Aws amity university_presentation_oct-31-2020
Aws amity university_presentation_oct-31-2020
 
Webcast: DevOps in AWS is different! How can containers help?
Webcast: DevOps in AWS is different! How can containers help? Webcast: DevOps in AWS is different! How can containers help?
Webcast: DevOps in AWS is different! How can containers help?
 
DevOps Columbus Meetup Kickoff - Infrastructure as Code
DevOps Columbus Meetup Kickoff - Infrastructure as CodeDevOps Columbus Meetup Kickoff - Infrastructure as Code
DevOps Columbus Meetup Kickoff - Infrastructure as Code
 
Containers, microservices and serverless for realists
Containers, microservices and serverless for realistsContainers, microservices and serverless for realists
Containers, microservices and serverless for realists
 
"Portrait of the developer as The Artist" Lockheed Architect Workshop
"Portrait of the developer as The Artist" Lockheed Architect Workshop"Portrait of the developer as The Artist" Lockheed Architect Workshop
"Portrait of the developer as The Artist" Lockheed Architect Workshop
 

Recently uploaded

UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
Mariano Tinti
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
SitimaJohn
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
David Brossard
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
Zilliz
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 

Recently uploaded (20)

UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 

How to run a global, cloud scale event for 10.000 people

Editor's Notes

  1. We started in New Zealand Stopped in Seattle
  2. You just saw the intro video for the global devops bootcamp 2019. A community event we organized for the last three years. We want to tell you this story on how we run this over 35 countries with 10000 participants, 7 azure devops environment and causing 3 outages in Azure DevOps.
  3. Who are you doing it for / with? Vision Working together in a globally distributed team Open source / daily job Clearly defined purpose Empowerement Isolated architecture as a starting point Communication Technical design decisions No big design up front Do think big Scalable Monetary restrictions Techical restrictions Design with the end in mind
  4. 2017: What is DevOps 2018: DevOps at Microsoft 2019: SRE & DevOps
  5. Team of volunteers to create content Volunteers Sponsoring in time/money/Azure Credits/Tweet wall/Snyk package scanner support Planning: Weekly meeting in MS Teams
  6. Message from the CEO of Parts Unlimited
  7. SWITCH SPEAKER What do we need to provision in Azure
  8. Goal is a working webshop for a team of attendees Webshop: App Service / Sql DB / Application Insights Deliver working webshop for attendees + CI/CD pipeline in Azure DevOps
  9. Webshop has been selected Team of 5 attendees
  10. Started with Azure CLI Switched to ConsoleApp.exe Scalable, Repeatable process  Azure DevOps
  11. Our own pipeline Azure Separated for fast iterations during preparation phase
  12. SQL Server: 20 is a soft limit: resolved through support ticket (two days to late!) 1200 / 4 subs = 300 SQL Servers…. No interesting limits on e.g. App Service Plans (100 per resource group). 200 is a hard limit
  13. Crucial factor, Credit Card
  14. Inception! What now? Rolling out webshops to production 1 Azure DevOps Project to provision Azure DevOps team projects (x1200)
  15. SWITCH SPEAKER
  16. Per team needed
  17. 7 Azure DevOps organizations: one for each supported region All attached to same AAD: so one account to rule them all Concurrent hosted pipelines: 100 or 150 per organization: 200 teams per organization
  18. 7 Azure DevOps organizations: one for each supported region All attached to same AAD: so one account to rule them all Concurrent hosted pipelines: 100 or 150 per organization: 200 teams per organization Peak usage: 700 concurrent pipelines
  19. Full pipeline overview
  20. Certificate separate : timing issue with App Service Ready
  21. Part 1: Export dataset = ConsoleApp.exe Provision Azure DevOps team projects + AD DNS here, because it takes a while to be ready Part 2: Init AzDo Team Project Part 3: Trigger all the builds (would incur cost)
  22. Part 1: Export dataset = ConsoleApp.exe Provision Azure DevOps team projects + AD DNS here, because it takes a while to be ready Part 2: Init AzDo Team Project Part 3: Trigger all the builds (would incur cost)
  23. Part 1: Export dataset = ConsoleApp.exe Provision Azure DevOps team projects + AD DNS here, because it takes a while to be ready Part 2: Init AzDo Team Project Part 3: Trigger all the builds (would incur cost)
  24. Part 1: Export dataset = ConsoleApp.exe Provision Azure DevOps team projects + AD DNS here, because it takes a while to be ready Part 2: Init AzDo Team Project Part 3: Trigger all the builds (would incur cost)
  25. Build pipeline outage in 2 regions: starting 400 pipelines (with 400 CD releases after them!) causes some load on Azure DevOps Scaled down to 1 concurrent pipeline on all regions Austalia: Networking issue on Scaleset  Npm restore failed Rene in call until 12 PM
  26. This is the break
  27. SWITCH SPEAKER
  28. SWITCH SPEAKER
  29. Switch to gdbc-challenge-com Secured by Azure AD integration Table storage for team state
  30. ACI: nice, no orchestration needed, might be cheaper ACI: to slow to start a container ACI Constraint: max 300 container create per hour
  31. Google Analytics AppIication Insights
  32. Checking the load on the cluster – Region Europe We see that pods are starting, stopping isn’t that visible We use short lived pods, but the default garbage collection is on 12.000 pods, so that line doesn’t go down
  33. Custom events with tracking ID
  34. Slack channel + Bridge on Teams Slack channel #SRE Bridge on Microsoft Teams Command Center
  35. Monitoring, be in control Fallbacks, caching and backups
  36. Preparation is key Support tickets for raising SQL Servers limits took 2 weeks and where to late! Twitter preparation Event starts in New Zealand: that is 11 PM for us in NL New Zealand: 11:00 PM Started Europe: 09:00 AM West coast US ended at 02:00 AM