1.
Load Balancing
in the SRE way
Ke Zhu
@shawnzhu
Site (Un)Reliability Engineer at IBM

2.
For What?
• GitHub Enterprise Cluster
• On Internet
• Zero downtime
• 100M+ HTTP requests per week
• 30k+ attacks per week
• 26k+ git clone per hour
(https://help.github.com/enterprise/2.8/admin/guides/installation/maintenance-mode/)

3.
Design Goals
• Scripting Platform
• traffic conducting via code
• do social coding
• Observable
• Blue/Green deployment
• High performance
• Security from day one

4.
Software Stack
&& 🎩 magic kernel parameters in /etc/sysctl.conf 🐰

5.
Scripting Platform
• OpenResty (Nginx + Lua) - https://openresty.org/en/
(Example: customized request rate limiting)

6.
Blue/Green Deployment
• Can not terminate any TCP connection
• Two stacks:
• load-balancer-green
• load-balancer-blue (for experiment)
• Cloud DNS
• Switching A record + short TTL (~5m)
• Simple/Weighted Routing policy
• Run experiment by using docker image tags
• Real time metrics collection by librato.com

7.
• Test docker images
• RSpec + Serverspec
• Travis CI
• Test docker host
• RSpec + Serverspec
• Test Kitchen
Test Driven for Container

8.
❤vault
• Secret mgmt via API - https://www.vaultproject.io/
• retrieve all secrets for provisioning load balancer
via a single token with TTL 5min

9.
Blocking mode in Production
• Signal Sciences - https://signalsciences.net/

10.
Summary
• Conducting HTTPS traffic via Lua code
• Blue-green deployment of Load balancer via DNS
• Testing docker with RSpec + Serverspec
• SignalSciences