The Stack Exchange infrastructure supports 560 million page views and 34TB of data transferred per month across multiple technology stacks and datacenters. Performance is the top priority, and tools like Mini Profiler, OpServer, and Client Timings are used to monitor and improve performance. The infrastructure is designed with redundancy across networks, load balancers, web and database servers, caching, and search to ensure high availability and fast response times below 60ms for core pages.

1. The Stack Exchange
Infrastructure
Vroom Vroom

2. inet.perf.profile
• SRE Generalist @ Stack Exchange
• @GABeech
• http://brokenhaze.com
• http://stackexchange.com

3. A brief Overview
• 560 Million Page Views a Month
• 34TB of Data transfered a Month
• 1665 rps (2250 peak) Across web Farm
• WISC(HER)
Windows
IIS
SQL Server
C#
HAProxy
Elastic Search
Redis

4. Our First Priority is
Performance
Nobody likes a slow site, least of all us.
When your site is slow people leave.
!
Make your site fast, and the people will stay
!
Good write up on moz.com:
http://moz.com/blog/site-speed-are-you-fast-does-it-matter
Why do I bring up performance in an infra talk? simple. It drives our design decisions.

5. The Performance
toolkit
• Mini Profiler
• OpServer (https://github.com/opserver/
Opserver)
• Client Timings (http://
teststackoverflow.com/)

6. Mini Profiler
Shown to every Dev/SRE on every page
Oneboxed in our chat system

7. OpServer
Bubbles up problems

8. OpServer HAproxy

9. OpServer Redis

10. OpServer SQL

11. Client Timings
How well are we actually doing when _you_ load the page

12. You can’t be fast if you
are not up
• Highly Redundant network
• Datacenter, ISP, Edge, Core, Server, Port
The actual design starts now.

13. 4 Different providers
Selected for different characteristics
Router Redundancy Hot/Standby HSRP/BGP on “T2”
Full BGP tables and HSRP on T1

14. Load Balencers
• HAProxy
• 2 Servers (Hot/Standby)
• Multiple Tiers (HAProxy Processes)
4B requests/month
3000 req/sec peak
10% CPU 18% peak
Between 600 and 700 concurrent connections (EST, TIME_WAIT, ETC)
Multiple Processes Allow for granular restarts and segregation of faults
SSL Termination done on the LB
Websockets: The weird connection
Long lived
TCP not HTTP

15. Request flow
In, is http? yes, servers: no term https, is http

16. SSL Termination
• Terminated at LB
• Feature added to HAProxy 1.5
• See: http://brokenhaze.com/blog/
2014/03/25/how-stack-exchange-gets-the-most-
out-of-haproxy/
Source Port Exhaustion
use 127.0.0.0/8 to resolve
Server only running at ~12% cpu
We don’t run full SSL everywhere yet

17. Web Servers
!
• IIS
• 9 Production (2 Test/Dev)
• Dell R610’s
• 32GB Memory
• 2xE5-5640
185 req/s 250 peak
15% CPU usage 20% peak

18. Data Tier
• MS SQL Server
• 4 Servers
• 2 Always-On Clusters
• Each Cluster 1 RW, 1 RO
(SO) 343 M Queries per day
(SO) Peak of 7500 queries / second
(SE) 216M Queries per day
(SE) Peak 3200 queries / second
!
CPU Use: SO 8% Peak 15% — SE 10% Peak 20%

19. Caching Tier
• Redis
• 2 Servers
• Hot / Standby configuration
3.65 B operations a day
Peak 60,000/s
3% cpu usage
!

20. Tag Engine
• Our Special index of SO
• Tagging is hard
• Written by Marc Gravell
• http://blog.marcgravell.com/2014/04/technical-debt-
case-study-tags.html
3 Servers, 32 GB RAM
3644 req/s
3% CPU 10% peak
Replaced Full Text search in SQL Server
Spins up a full copy of SO/SE
Cool thing can be upgraded with 0 downtime

21. Elastic Search
• 203GB Index
• 3 Machines
• 42M searches/day
2 others/ not prod
Machine learning
Log stash (300TB)

22. Deployment
• Git
• TeamCity
• Custom Powershell Scripts
Team City monitors our Development Git repository
Dev Auto builds (Deploy to Meta)
When the build is verified Dev triggers Prod Build
Copy Artifacts from Dev Build

23. So what does this get
you
• 52 ms homepage render time
• 33 ms questions page render time

24. Always See our
Performance
• http://stackexchange.com/performance

25. Thank YOU!
Contact:
@GABeech
george@stackoverflow.com
Office Hours:
Wednesday, November 12th
(today…)
2:00pm - 3:30pm
LISA Lab