This document summarizes Squarespace's transition from a monolithic architecture to a microservices architecture and their implementation of a service mesh using Envoy proxy. It describes how Squarespace grew from less than 50 engineers in 2013 to over 200 engineers in 2017, necessitating the move to microservices for scalability. It outlines their initial use of Consul for service discovery and Netflix OSS libraries. It then introduces the concept of a service mesh and how Envoy proxy deployed as a sidecar can provide advanced control, observability, and support for multiple languages. It details how Envoy uses Consul and its xDS APIs for dynamic service discovery and configuration. Finally, it discusses future work including integrating orchestration and abstracting common service

1. Building a Service Mesh with Envoy
Doug Jones
djones@squarespace.com
@dougfjones

2. Microservices: A Story of Growth
Monolith
Background Jobs
DBQueue
2013: <50 engineers
● “Whatever works”
● Build product
● Grow fast

3. Microservices: A Story of Growth
2014: ~75 engineers
● “Whatever works”
● Too much firefighting
● Not enough new features
● Inflexible monolith architecture
Monolith
Background Jobs
DBQueue

4. Microservices: A Story of Growth
Monolith
Background Jobs
DBQueue
2016: 100+ engineers
● Microservices
● Scalable + Reliable
● Developers can move faster
● Squarespace can move faster

5. Microservices: A Story of Growth
Monolith
Background Jobs
DBQueue
2017: 200+ engineers
● Even More Microservices
● Independent, full stack teams
● Self Service Infra with
Kubernetes
● Desire flexible, reliable infra
● Desire better tooling

6. Microservices at Squarespace
● Building started late 2014
● Java API Servers
● Virtual Machines
● Consul Service Discovery
● Service client based on Netflix OSS (Hystrix, Ribbon, RxNetty)
● Now in progress: migrating from VMs to Kubernetes
Microservices Platform

7. Consul Use Case
Consul
Service A
Service B
Client Lib
Service B
Request
Service
Instances
Load Balance
Announce
Presence

8. DC 2DC 1
Consul Cross DC
Consul
Service A
Service B
Service B
Consul
Service C
Service D
Service D
Cross DC Gossip

9. Service Mesh
● Service client functionality moves to its own process (sidecar)
○ No longer trapped in a library
● This process can be configured and updated independently of the
application it serves
● Advanced operational control through APIs
● Improved observability
● Opens the door to better support for service development in multiple
programming languages
Why Service Mesh?

10. Service Mesh
● Envoy proxy
○ Co-located with each service instance (sidecar)
○ Proxies ingress and egress traffic
● Dynamic configuration API
○ Provide service discovery information
○ Change routing table and circuit breaker configuration
○ Big upgrade in our capability as operators
Service Mesh with Envoy

11. Pod
Envoy and Consul
Consul
Service A
Service B
Service BLoad Balance
Announce
PresenceEnvoy
Mesh
Discovery
Streaming
Updates (v2
API)

12. Envoy V2 xDS Proto
ConsulEnvoy
Mesh
Discovery
Cluster
Discovery
(CDS)
Catalog List
Services

13. Envoy V2 xDS Proto
ConsulEnvoy
Mesh
Discovery
[“Service A”,
“Service B”]
VersionInfo:
123
[“Service A”,
“Service B”]
Index: 123

14. Envoy V2 xDS Proto
ConsulEnvoy
Mesh
Discovery
Catalog List
Services
Newer than
Index: 123
Poll Consul

15. Envoy V2 xDS Proto
ConsulEnvoy
Mesh
Discovery
[..., “Service
C”]
Index: 456
[..., “Service
C”]
VersionInfo:
456
Push Update

16. Envoy xDS Proto
● EDS -> Consul health endpoint list Nodes for Service
○ Uses same Consul index to version mapping
○ Background polling for updates
● RDS
○ Simple HTTP routing rule matching header value to cluster name
Other xDS Requests

17. Observability

18. Future
● Build orchestration features into our new discovery system
○ Harness xDS to push updates that make traffic routing changes
● APIs to abstract common service mesh operations
● Dashboard for operators
Future Work

19. Thank You!
squarespace.com/careers
Doug Jones
djones@squarespace.com
@dougfjones