Running e-Commerce at Scale
Presenter
Ian Bissett
Date
September 12, 2017

Who am I ?
Manager, Site Reliability Engineering @
BigCommerce
Over 17 years scaling big web infrastructure:
- BigCommerce
- Fairfax Media - smh.com.au, theage.com.au
- BBC - bbc.co.uk - BBC iPlayer
Now:
- Geek
- Pirate
- Fire Fighter with the RFS
https://www.linkedin.com/in/bisscuitt
@bisscuitt

PHP from an Operational
point of view
- PHP in production
- Getting Requests to the hosts
- Asset storage at scale
- Database sharding
- Caching, caching, caching
- Load Balancing
- Service Discovery
Files, Images
& Templates
L4 Load Balancers and
Caching Layer
Application Servers
Store Config
and Caching
Orders and
Product Metadata
L3 Load Balancers
LVS
Database
Servers
Asset and File
Storage
Configuration and
Caching Service

Application Servers
bcserver
LUA
Shipping Service
Config Service
BigCommerce App
Twemproxy

Admin ServerRead Only Slave
Active Master Backup Master
Replication
Slave Only
Database Clusters

Asset Storage
Load Balancer Load Balancer
Mule Host Mule Host

Volatile Redis
Twemproxy (aka nutcracker):
https://github.com/twitter/twemproxy
App Servers
pool_1:
listen: 127.0.0.1:6379
hash: crc32a
distribution: ketama
redis: true
auto_eject_hosts: true
servers:
- 10.1.1.1:63791:100 redis-11
- 10.1.1.1:63792:100 redis-12
- 10.1.1.1:63793:100 redis-13
- 10.1.1.1:63794:100 redis-14
- 10.1.1.2:63791:100 redis-21
- 10.1.1.2:63792:100 redis-22
- 10.1.1.2:63793:100 redis-23
- 10.1.1.2:63794:100 redis-24

Persistent Redis
- Many single threaded instances of redis run on
each redis hosts
- Instances configured in master -> slave pairs
- All Redis sentinels monitor the state of the redis
instances.
- Redis Sentinels keep state and form a quorum
- Apps connect to redis sentinel to request
address of current master
- App then connects directly to redis instance
Permanent Redis Servers
Redis Sentinels
Redis Master Redis Slave
Replication
Sentinel 2 Sentinel 3
Sentinel 1
Healthchecks
require 'Predis/Autoloader.php';
PredisAutoloader::register();
$sentinels = ['tcp://10.0.0.1', 'tcp://10.0.0.2', 'tcp://10.0.0.3'];
$options = ['replication' => 'sentinel', 'service' => 'mymaster'];
$client = new PredisClient($sentinels, $options);

L4 Load Balancing
LUA
Asset
Cache
Client Request
Store
Config
Lookups
GenerateConfig
Service Status Sync
CDN
Asset Request
Asset Storage
Fetch uncached Assets
ssl_certificate_by_lua
App Servers

Layer 3 (IP) Load Balancing
BGP
Announcements BGP
Announcements
L4 Load Balancers
Firewall
Pair 1
Firewall
Pair 2

Tech I haven't even mentioned yet!

Thanks!
Questions?!
Ian Bissett - @bisscuitt
ian.bissett@bigcommerce.com
Files, Images
& Templates
L4 Load Balancers and
Caching Layer
Application Servers
Store Config
and Caching
Orders and
Product Metadata
L3 Load Balancers
LVS
Database
Servers
Asset and File
Storage
Configuration and
Caching Service