Site Reliability Engineering Portland Meetup May 2017

1. Site Reliability Engineering Portland
Building an SRE Organization
Franklin Angulo
@feangulo

2. 01 History
02 Bootstrapping an SRE Team
03 Growing an SRE Organization
04 SRE Organization at Squarespace
05 Questions
AGENDA

3. History

4. Squarespace

5. New York City

6. Portland

7. Dublin

14. The Beginning
Server Systems Product & Data

15. The Beginning

16. The Beginning

19. NEW YORK CITY

21. SkipRead
30 5
Alarm
9
Alerting Math

23. Publish
Metrics
Store
Metrics
Build
Dashboard
Identify
Patterns
Create
Monitors
1 2 3 4 5
Workflow

24. Workflow

28. Graphite replicated cluster
800K metrics per minute
158 Grafana dashboards

30. Bootstrapping an SRE Team

31. New SRE Team
Server Systems Product & DataSRE

32. Three Pillars of SRE
Monitoring &
Alerting
Configuration
Management
Builds & Deploys

33. Configuration Management

34. Provisioning
Requirements:
● Cobbler
● IP Plan
● DNS
● LDAP
● YUM
● NTP
● ESXi
● vCenter
Provisioning Steps:
1. Pick an IP and server with resources
2. Connect to vCenter, create VM, get macaddr
3. Add system to Cobbler
4. Apply the base role
Challenges:
● VM vs. physical server?
● What is included in the base role?
● Which IP is available?
● Which server has enough resources?

35. Configuration Management
2013 20162014 2015
First commit!
October 17, 2013
2017

36. 2013 20162014 2015
111 commits, 2 contributors
January 1, 2014
2017
Configuration Management

37. 2013 20162014 2015
2662 new commits, 20 contributors
January 1, 2015
2017
Configuration Management

38. 2013 20162014 2015
5799 new commits, 38 contributors
January 1, 2016
2017
Configuration Management

39. 2013 20162014 2015
6059 new commits, 56 contributors
October 10, 2016
2017
Configuration Management

40. Configuration Management
Playbooks Roles
Inventory

41. Configuration Management
Playbooks Roles
Inventory

42. Configuration Management
Playbooks Roles
Inventory
~300 ~200
~2200 hosts

43. Provisioning
Inventory Entry:
Playbook Execution:

44. Collectd
MongoDB
Cassandra
MySQL
Kubernetes
PostgreSQL
Mesos
Nginx
Kafka Elasticsearch
Sentry
Boulder CA
Cobbler
Docker
Registry
Grafana
Discourse
Redis
Zookeeper
Graphite
Bitbucket
Sensu
Confluence
Consul
Cloudera HDFS
Couchbase

45. Builds & Deploys

46. Builds & Deploys

47. Builds & Deploys

48. Builds & Deploys

49. Builds & Deploys

50. Builds & Deploys

51. Monitoring & Alerting

52. Monitoring & Alerting

53. Monitoring & Alerting

54. Monitoring & Alerting

55. Monitoring & Alerting

56. Monitoring & Alerting

57. Monitoring & Alerting

58. Operations & On-Call

59. Operations & On-Call

60. Operations & On-Call

61. Operations & On-Call

62. SRE Team (2 Years)
Server TechOpsSRE

63. Growing an SRE Organization

64. SRE Team (Present)
Server
TechOps
SRE

65. TechOps
Site Reliability Engineering (SRE)
Edge Infrastructure Service Infrastructure
Application Infrastructure Core Services
Server Engineering Organization

66. TechOps
Site Reliability
Embeds
Edge Infrastructure Service Infrastructure
Application Infrastructure Core Services
Server Engineering Organization
Site Reliability
Operations
Provisioning
Release
Engineering /
Developer
Productivity
Observability
& Alerting

67. Site Reliability Embeds

68. Site Reliability
Embeds
Edge Infrastructure Service Infrastructure
Application Infrastructure Core Services
Site Reliability Embeds

69. Provisioning

70. Provisioning
Provisioning
● Cobbler
● IP Plan
● DNS
● LDAP
● YUM
● NTP

71. Release Engineering &
Developer Productivity

72. Release
Engineering /
Developer
Productivity
Release Engineering & Developer Productivity
Drone
Helm

73. Observability & Alerting

74. Observability
& Alerting
Observability & Alerting

75. Squareflux

76. Statistics
Observability & Alerting
Log Lines
13 MM
per minute
Metrics Dashboards Healthchecks
442 7,429
2016
2017
4.1 MM
per minute
300,000
per minute
282 -
1,000,000
per minute

77. Site Reliability Operations

78. Site Reliability
Operations
Site Reliability Operations
Provisioning
Release
Engineering &
Developer
Productivity
Observability
& Alerting
Site Reliability
Embeds

79. Site Reliability
Operations
Site Reliability Operations
Site Reliability
Embeds
Development
Teams
Production Readiness Scorecard

80. SRE Organization at Squarespace

81. SRE Team (Growth)
Portland New York Dublin
Site Reliability Embeds Provisioning / Release Engineering / Observability / Alerting Site Reliability Operations

82. Squarespace SREs

83. Franklin Angulo
@feangulo
Thank you!
www.squarespace.com/careers