The document discusses the infrastructure and technologies behind Wordpress.com, highlighting its extensive use of databases, caching, and content delivery systems to ensure scalability and performance. It emphasizes the importance of tools like HyperDB for database management, advanced caching mechanisms, and a focus on security through strategies like DDoS protection and SSL implementation. The final section suggests best practices such as sharding databases, caching at the edge, and employing AI for resource planning.

1. A Tale of Two Systems
David Newman
@darthhexx

2. Howdy! We're the people behind
WordPress.com, WooCommerce,
Jetpack, and a bunch of other
products for WordPress.

3. We are passionate about
making the web a better place.
We don't build software for free
– we build it for freedom.

5. Communication is oxygen

6. A TALE OF TWO SYSTEMS

8. WordPress.com
WordPress multisite with 100+ million sites.
VIP Go
Container-based VIP WordPress hosting.

9. A TALE OF TWO SYSTEMS
WordPress.com

10. Simplicity scales.

11. WordPress.com
• The App
• Database
• Static content
• Caching
• Distributing workload

12. WordPress.com
• Networking
• Protecting
• Continuous Integration
• Stats and analysis

13. The App

14. • Free Open Source Software.
• Most widely adopted server-
side programming language.
• Notable performance increase
in PHP7.

15. • Free Open Source Software.
• 27% of top 10 million sites.
• Passionately developed and
supported.

16. • Add/remove/apply filters
and actions, offers
extensibility.
• We customise, add features
and improve performance.

17. • HyperDB
• Object Cache
• CSS/JS concatenation
• 2FA
• many, many more…
WordPress.com

19. Databases

20. Databases
A billion+ tables

21. HyperDB
• Slave lag and failed host
detection and mitigation.
• Distributed reads / writes.
• Replication and partitioning
support.

22. Partitioning
• Global tables for sites, blogs,
users, usermeta, etc.
• Sharded blog specific data
(posts, comments, options,
terms, etc.)

23. Databases
• Query comments containing
the URL in order to track down
errant code.
• Dedicated replicated read-only
DBs for backups.

24. Databases
• “Index all the things” and
EXPLAIN everything :)
• Use MyISAM, unless workload
warrants the use of InnoDB,
due to memory requirements.

25. Static Content

26. Life Without Static Content

27. Images
• Distributed fault tolerant file
system with MogileFS.
• Clusters in multiple
Datacenters.
• Replicated to S3 as immutable
backups.

28. Images
• Do not store intermediate
sizes.
• Resize images on the fly.
• Responsive design with srcset.

29. Images
• Perform image optimisations
with pngquant, optipng, and
jpegoptim.
• Output progressive JPEGs.

30. Images
• On-the-fly WebP conversions
based on HTTP Accept header.
• Disable advanced compression
routines under load.

31. CSS/JS
• Dedicated static content
production servers.
• Minified at commit.
• Concatenated on the fly in
production.

32. Caching

33. Cache Levels
• Server-side
• Datacenter
• PoP / Edge

34. PHP OpCache
• Compiled PHP scripts cached.
• Linked into our deploy system
to auto-compile / delete as
necessary.

35. PHP APCu
• Great for “persistent statics”.
• Querying load in order to
switch off non-critical image
optimisations.
• File system IO processes for
consistently hashed services.

36. Datacenter Cache
• Distribution requires shared
caches.
• Object Cache shards cache
groups.

37. Memcached
• Drop in Object Cache
replacement: wordpress.org/
extend/plugins/memcached/
• Internal systems for sync’ing
DC caches.

38. • Multi-DC replication challenge
with DB vs Cache replication
timing.
• Sync’ing cache invalidations
with DB replication, e.g.
Facebook’s mcrouter
Memcached

40. Batcache
• Uses Memcached to store and
serve rendered pages:
wordpress.org/plugins/
batcache/
• 40x reduction in page
generation time.

41. NGiNX DC Cache
• Caching of some assets at the
DC level, e.g. feeds, sitemaps,
some images, etc.

43. PoP / Edge Caching
• Every “corner” of the world.
• Only cache after more than X
page queries in Y time period.
• OpenResty for advanced
features.

44. PoP/Edge Invalidations
• Performed using mangle, an
internally developed tool.
• Employs Anycast, using
encrypted UDP, and a gossip
protocol.

45. Distribute

46. Distribute
Use the right tool for the job at hand.

47. Async Jobs System
• WordPress-based jobs system.
• Time insensitive processes
performed asynchronously.
• wp-cron jobs, do_pings, etc.

48. • “Index all the things”.
• Used for performant full text
searches on content.

49. • Centralised application log
storage.
• Logs shipped via Logcourier.
• Kibana for dashboards and
alerts.

50. Networking

52. Networking
• HA via Anycast.
• Custom server ToR setup using
VRRP.
• Production traffic routing.

53. Unicast

54. Broadcast

55. Multicast

56. Anycast

57. Anycast

58. Anycast
• BGP advertises IP address
subnets between networks.
• Local preference metric, which
overrides the network hop
metric (AS Path), adds peering
complications.

59. Anycast

62. Anycast Benefits
• Distributed caches.
• Low network latency to closest
PoP.
• Easier to perform maintenance.
• DDoS mitigation.

63. 170Gbps DDoS Absorption

64. Custom ToR Setup
Not this ToR :)

65. • Anycast between ToR and
cores.
• VRRP using the host as the
bridge between the 2 ToR
switches.
Custom ToR Setup

66. • VRRP is active/passive virtual
address that can exist on one
of 2 devices.
• VRRP master used as the host
gateway.
Custom ToR Setup

67. • Multichassis Link Aggregation
(MLAG) to form 1 LACP port
channel.
• Provides active/active on
Layer 2.
Custom ToR Setup

68. Custom ToR Setup
Host
Switch BSwitch A
DC
BGP
VRRP
MLAG / LACP
VRRP
MLAG / LACP
BGP

69. • IP address allocation is
efficient.
• Sub-second failover.
• Easier switch maintenance.
• L3 straight to the server, L2
domain is the server itself.
Custom ToR Setup

70. • Route certain endpoints and
content types to specialised
backends, e.g. API, wp-admin,
statics, etc.
• Global traffic balancing using
NGiNX split_clients.
Production Traffic Routing

71. Protecting

72. If you build it, they will come.

73. • Anycast absorbs DDoS by
design.
• SSL everywhere using Let’s
Encrypt.
Protecting

75. • IPSET auto-blocking
mechanisms.
• OpenResty for SSL blocking
algorithms in NGiNX.
Protecting

76. Comment SPAM

77. Akismet
Filters for spam comments on
millions of sites in real time.

78. CI

79. CI

80. • Each engineer has one.
• Production test/staging sites.
• Huge advantage for production
debugging.
• Integrated deploy tests and
validations.
Sandbox

81. • Parallel tests using Agents.
• Supports PHPunit tests.
• Local sandboxed test data.
TeamCity

82. Stats and
Analysis

83. • Tracks for user events.
• Sqoop’d into Hadoop and
accessed via Hue2.
• Cloudera, Impala, Zookeeper,
etc.
Stats and Analysis

84. A TALE OF TWO SYSTEMS
VIP Go

85. Outline
• Shared infrastructure; isolated
client platform.
• No restrictions set on plugins.
• Seamless deploy to Dev, Test,
Staging and Production.
• Integrated review process.

86. chroot > solution < VM

87. Containers

88. Docker or pure LXC?

90. Helicopter View
• Container-based hosting
platform.
• Internal Docker Registry.
• Custom container orchestration
via Host Action queues.
• API driven.

91. Similarities
• Static content
• Protecting
• Distributing workload

92. Distinctions
• The App Containers
• Networking
• Caching
• Continuous Integration
• Auto-scaling

93. App Containers

94. App Containers
• Web and DB containers use
Docker as one would LXC, with
`/sbin/init`.
• Memcached Docker-style.
• AUFS only.
• Pin to Docker version.

95. App Containers
• DB containers are InnoDB only.
• Percona XtraBackup tools.
• Web Containers have Jetpack
Premium installed.
• New Relic real time PHP
analysis.

96. Networking

97. Networking
• Avoid using the NAT userspace
docker-proxy where possible.
• Use “host-only” wherever
possible.
• Dynamic service port assignment
via container orchestration.

98. Networking
• Dynamic service port firewall in
the containers.
• Set ephemeral port range in
net.ipv4.ip_local_port_range.
• Watch the conntrack table!

99. Caching

100. Caching
• Varnish caches all assets at the
Edge.
• Be weary of `Hit-for-pass`.
• Websockets need special
configs and attention.

101. Caching
• Varnish and TLS, the short
story.
• Leap second, design for failure.

102. Continuous Integration

103. Continuous Integration
• Git-centric (Github / Gitlab).
• Clients select their own CI / Test
toolchain.
• Review Queue integration.

104. Continuous Integration
• Dev / Staging / Production tied
to repo branches.
• Host Action controlled code
deploys triggered by
WebHooks.

105. Stats and Auto-scaling

106. Auto-scaling
• InfluxDB timeseries stats.
• API auto-allocates containers to
hosts using signals from host
stats.

107. Auto-scaling
• Auto-scaling service
interrogates InfluxDB stats and
scales reactively.
• Predictive resource allocation
using Machine Learning.

108. Summary
• Shard DBs or Clients :)
• Cache at the Edge.
• Anycast whenever you can.
• Employ simple active defences.

109. Summary
• Design for failure.
• Sync invalidations with data
streams.
• Employ AI for resource
planning.

110. Simplicity scales :)

111. Q & A

112. Thanks
David Newman
@darthhexx