Site Reliability Engineering Portland
Building an SRE Organization
Franklin Angulo
@feangulo
01 History
02 Bootstrapping an SRE Team
03 Growing an SRE Organization
04 SRE Organization at Squarespace
05 Questions
AGENDA
History
Squarespace
New York City
Portland
Dublin
The Beginning
Server Systems Product & Data
The Beginning
The Beginning
NEW YORK CITY
SkipRead
30 5
Alarm
9
Alerting Math
Publish
Metrics
Store
Metrics
Build
Dashboard
Identify
Patterns
Create
Monitors
1 2 3 4 5
Workflow
Workflow
Graphite replicated cluster
800K metrics per minute
158 Grafana dashboards
Bootstrapping an SRE Team
New SRE Team
Server Systems Product & DataSRE
Three Pillars of SRE
Monitoring &
Alerting
Configuration
Management
Builds & Deploys
Configuration Management
Provisioning
Requirements:
● Cobbler
● IP Plan
● DNS
● LDAP
● YUM
● NTP
● ESXi
● vCenter
Provisioning Steps:
1. Pick an IP and server with resources
2. Connect to vCenter, create VM, get macaddr
3. Add system to Cobbler
4. Apply the base role
Challenges:
● VM vs. physical server?
● What is included in the base role?
● Which IP is available?
● Which server has enough resources?
Configuration Management
2013 20162014 2015
First commit!
October 17, 2013
2017
2013 20162014 2015
111 commits, 2 contributors
January 1, 2014
2017
Configuration Management
2013 20162014 2015
2662 new commits, 20 contributors
January 1, 2015
2017
Configuration Management
2013 20162014 2015
5799 new commits, 38 contributors
January 1, 2016
2017
Configuration Management
2013 20162014 2015
6059 new commits, 56 contributors
October 10, 2016
2017
Configuration Management
Configuration Management
Playbooks Roles
Inventory
Configuration Management
Playbooks Roles
Inventory
Configuration Management
Playbooks Roles
Inventory
~300 ~200
~2200 hosts
Provisioning
Inventory Entry:
Playbook Execution:
Collectd
MongoDB
Cassandra
MySQL
Kubernetes
PostgreSQL
Mesos
Nginx
Kafka Elasticsearch
Sentry
Boulder CA
Cobbler
Docker
Registry
Grafana
Discourse
Redis
Zookeeper
Graphite
Bitbucket
Sensu
Confluence
Consul
Cloudera HDFS
Couchbase
Builds & Deploys
Builds & Deploys
Builds & Deploys
Builds & Deploys
Builds & Deploys
Builds & Deploys
Monitoring & Alerting
Monitoring & Alerting
Monitoring & Alerting
Monitoring & Alerting
Monitoring & Alerting
Monitoring & Alerting
Monitoring & Alerting
Operations & On-Call
Operations & On-Call
Operations & On-Call
Operations & On-Call
SRE Team (2 Years)
Server TechOpsSRE
Growing an SRE Organization
SRE Team (Present)
Server
TechOps
SRE
TechOps
Site Reliability Engineering (SRE)
Edge Infrastructure Service Infrastructure
Application Infrastructure Core Services
Server Engineering Organization
TechOps
Site Reliability
Embeds
Edge Infrastructure Service Infrastructure
Application Infrastructure Core Services
Server Engineering Organization
Site Reliability
Operations
Provisioning
Release
Engineering /
Developer
Productivity
Observability
& Alerting
Site Reliability Embeds
Site Reliability
Embeds
Edge Infrastructure Service Infrastructure
Application Infrastructure Core Services
Site Reliability Embeds
Provisioning
Provisioning
Provisioning
● Cobbler
● IP Plan
● DNS
● LDAP
● YUM
● NTP
Release Engineering &
Developer Productivity
Release
Engineering /
Developer
Productivity
Release Engineering & Developer Productivity
Drone
Helm
Observability & Alerting
Observability
& Alerting
Observability & Alerting
Squareflux
Statistics
Observability & Alerting
Log Lines
13 MM
per minute
Metrics Dashboards Healthchecks
442 7,429
2016
2017
4.1 MM
per minute
300,000
per minute
282 -
1,000,000
per minute
Site Reliability Operations
Site Reliability
Operations
Site Reliability Operations
Provisioning
Release
Engineering &
Developer
Productivity
Observability
& Alerting
Site Reliability
Embeds
Site Reliability
Operations
Site Reliability Operations
Site Reliability
Embeds
Development
Teams
Production Readiness Scorecard
SRE Organization at Squarespace
SRE Team (Growth)
Portland New York Dublin
Site Reliability Embeds Provisioning / Release Engineering / Observability / Alerting Site Reliability Operations
Squarespace SREs
Franklin Angulo
@feangulo
Thank you!
www.squarespace.com/careers

Building an SRE Organization @ Squarespace