Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Scaling Stack Overflow
David Fullerton, VP Engineering • @df07
QCon NYC • 2015-06-12
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese an...
Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the sprea...
2
**SPOILERS**
3
Conclusions
1. Our architecture is boring
4
Conclusions
1. Our architecture is boring
1. How we keep it boring is interesting
5
What’s Stack Overflow?
6
Q&A for Programmers
• 9.4M questions
• 16M answers
• 45M uniques / month
• 8,000 new questions every day
(quantcast.com/...
7
Developer Jobs
• Best place on the internet to get a programming
job or hire a developer
8
Part of Stack Exchange Network
• Stack Overflow-style Q&A in 143 other topics &
languages
9
A Distributed Team
• 34 developers, 6 sysadmins, 6 designers
• 75% remote
10
A Distributed Team
• 34 developers, 6 sysadmins, 6 designers
• 75% remote
11
How do we work?
• Remote work culture
• Hire smart people and get out of their way
• Full-stack developers / sysadmins ...
12
Our Architecture
(I warned you, it’s boring)
13
(stackexchange.com/performance)
14
“Monolith Plus” architecture
• Almost everything happens in
the web tier + DB
• A few services pulled out and
optimized
15
Scales pretty well (for us)
• 4 billion requests per month, 3000 req/s peak
• 800M SQL queries per day, 8500/s peak
16
(opserver – https://github.com/opserver/opserver)
17
(opserver – https://github.com/opserver/opserver)
18
New York
(primary)
Oregon
(secondary)
Availability (also boring)
19
Deploys
• All day every day
• Rolling deploys through the web tier
(TeamCity)
Fast!
20
Testing
• Test on our users
• Feature flag
– Turn it on for a subset of sites to see how it
performs
21
* Works for us!
• Read-heavy load centered on one page
• Not as much customized content as some sites
• A forgiving com...
22
23
How did we get here?
24
Our Process
1. Start with what we know
2. Measure it live
3. Fix the slow
25
Step 1: Start with what we know
• Original developers knew C# and MSSQL
• Started with a bunch of off-the-shelf tools:
...
26
Step 2: Measure it live
• Performance is a feature!
• Test under real load
• Measure, don’t guess
27
(miniprofiler – https://github.com/MiniProfiler/dotnet)
28
(miniprofiler – https://github.com/MiniProfiler/dotnet)
29
(opserver – https://github.com/opserver/opserver)
30
(opserver – https://github.com/opserver/opserver)
31
Step 3: Fix the slow
• Slow performance is a bug, fix it now!
• Over time, replace major parts of our stack:
– Caching ...
32
• Already hand-rolling queries for performance
• LINQ to SQL provides basic ORM:
Dapper
33
• Problem:
Dapper
34
• Solution: replace the object mapper
• Idea: emit raw IL, then cache mapper
Dapper
35
• Results (500 iterations):
Dapper
(dapper– https://code.google.com/p/dapper-dot-net/)
36
Tag Engine
37
Tag Engine
• Early hack: use SQL fulltext search to index tags
38
Tag Engine
• Problem:
39
Tag Engine
• Problem:
• Performance!
40
Tag Engine
• Highly custom in-memory tag index cache
• Carefully memory-managed to avoid GC stalls
– Learned the hard w...
41
Results
42
Results
1. Start with what we know
2. Measure it live
3. Fix the slow
Optimize for performance,
get scale thrown in
43
Results
• “Monolith Plus” architecture
• Extract services that solve real problems, not
imagined ones
• Avoid SOA “tax”
44
So my primary guideline would be don’t even
consider microservices unless you have a
system that’s too complex to manag...
45
Conclusions
46
Conclusions
1. Our architecture is boring
2. How we keep it boring is interesting:
1. Start with what we know
2. Measur...
47
Application
• You can optimize for performance and get scale
thrown in (almost for free)
• Your monolith can scale furt...
48
Questions?
(We’re all about questions)
Obligatory:
• We’re hiring! stackexchange.com/work-here
• Open source! stackexch...
49
50
Here Be Dragons
(rejected slides)
51
• Started with basic OutputCache (cache rendered
HTML for a page)
• ~4% cache hit rate
Caching
52
• Add in-memory & Redis caching
Caching
53
StackExchange.Redis
• Wrote our own library for talking to Redis
• Multiplexing operations over a single connection
• A...
54
StackExchange.Redis
(opserver – https://github.com/opserver/opserver)
55
Moonspeak (Localization)
Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/stack-
exchange
Upcoming SlideShare
Loading in …5
×

of

Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 1 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 2 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 3 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 4 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 5 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 6 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 7 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 8 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 9 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 10 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 11 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 12 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 13 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 14 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 15 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 16 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 17 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 18 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 19 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 20 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 21 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 22 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 23 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 24 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 25 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 26 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 27 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 28 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 29 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 30 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 31 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 32 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 33 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 34 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 35 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 36 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 37 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 38 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 39 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 40 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 41 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 42 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 43 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 44 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 45 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 46 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 47 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 48 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 49 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 50 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 51 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 52 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 53 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 54 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 55 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 56 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 57 Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance Slide 58
Upcoming SlideShare
Mobile and Serverless : an Untold Story
Next

11 Likes

Share

Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1LjPgnq.

David Fullerton shares some of the things the Stack Exchange tech team have learned along the way while scaling one of the top sites in the world primarily through vertical scaling. Filmed at qconnewyork.com.

David Fullerton is the developer-turned-manager responsible for all software development and system administration at Stack Exchange. David joined Stack Exchange, Inc. from Fog Creek Software, where he worked briefly as the developer lead for Stack Exchange 1.0 and as a developer on the FogBugz team. He has a B.A. in Computer Science from Dartmouth College.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Scaling Stack Overflow: Keeping it Vertical by Obsessing Over Performance

  1. 1. 1 Scaling Stack Overflow David Fullerton, VP Engineering • @df07 QCon NYC • 2015-06-12
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /stack-exchange
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. 2 **SPOILERS**
  5. 5. 3 Conclusions 1. Our architecture is boring
  6. 6. 4 Conclusions 1. Our architecture is boring 1. How we keep it boring is interesting
  7. 7. 5 What’s Stack Overflow?
  8. 8. 6 Q&A for Programmers • 9.4M questions • 16M answers • 45M uniques / month • 8,000 new questions every day (quantcast.com/stackoverflow.com)
  9. 9. 7 Developer Jobs • Best place on the internet to get a programming job or hire a developer
  10. 10. 8 Part of Stack Exchange Network • Stack Overflow-style Q&A in 143 other topics & languages
  11. 11. 9 A Distributed Team • 34 developers, 6 sysadmins, 6 designers • 75% remote
  12. 12. 10 A Distributed Team • 34 developers, 6 sysadmins, 6 designers • 75% remote
  13. 13. 11 How do we work? • Remote work culture • Hire smart people and get out of their way • Full-stack developers / sysadmins with a specialty
  14. 14. 12 Our Architecture (I warned you, it’s boring)
  15. 15. 13 (stackexchange.com/performance)
  16. 16. 14 “Monolith Plus” architecture • Almost everything happens in the web tier + DB • A few services pulled out and optimized
  17. 17. 15 Scales pretty well (for us) • 4 billion requests per month, 3000 req/s peak • 800M SQL queries per day, 8500/s peak
  18. 18. 16 (opserver – https://github.com/opserver/opserver)
  19. 19. 17 (opserver – https://github.com/opserver/opserver)
  20. 20. 18 New York (primary) Oregon (secondary) Availability (also boring)
  21. 21. 19 Deploys • All day every day • Rolling deploys through the web tier (TeamCity) Fast!
  22. 22. 20 Testing • Test on our users • Feature flag – Turn it on for a subset of sites to see how it performs
  23. 23. 21 * Works for us! • Read-heavy load centered on one page • Not as much customized content as some sites • A forgiving community
  24. 24. 22
  25. 25. 23 How did we get here?
  26. 26. 24 Our Process 1. Start with what we know 2. Measure it live 3. Fix the slow
  27. 27. 25 Step 1: Start with what we know • Original developers knew C# and MSSQL • Started with a bunch of off-the-shelf tools: – ASP.NET MVC – LINQ to SQL – MSSQL + SQL fulltext search – Built-in caching (no Redis)
  28. 28. 26 Step 2: Measure it live • Performance is a feature! • Test under real load • Measure, don’t guess
  29. 29. 27 (miniprofiler – https://github.com/MiniProfiler/dotnet)
  30. 30. 28 (miniprofiler – https://github.com/MiniProfiler/dotnet)
  31. 31. 29 (opserver – https://github.com/opserver/opserver)
  32. 32. 30 (opserver – https://github.com/opserver/opserver)
  33. 33. 31 Step 3: Fix the slow • Slow performance is a bug, fix it now! • Over time, replace major parts of our stack: – Caching and Redis – SQL access – Tag Engine – Elasticsearch
  34. 34. 32 • Already hand-rolling queries for performance • LINQ to SQL provides basic ORM: Dapper
  35. 35. 33 • Problem: Dapper
  36. 36. 34 • Solution: replace the object mapper • Idea: emit raw IL, then cache mapper Dapper
  37. 37. 35 • Results (500 iterations): Dapper (dapper– https://code.google.com/p/dapper-dot-net/)
  38. 38. 36 Tag Engine
  39. 39. 37 Tag Engine • Early hack: use SQL fulltext search to index tags
  40. 40. 38 Tag Engine • Problem:
  41. 41. 39 Tag Engine • Problem: • Performance!
  42. 42. 40 Tag Engine • Highly custom in-memory tag index cache • Carefully memory-managed to avoid GC stalls – Learned the hard way: see “Assault by GC” by Marc Gravell • Serialize / deserialize from disk on build
  43. 43. 41 Results
  44. 44. 42 Results 1. Start with what we know 2. Measure it live 3. Fix the slow Optimize for performance, get scale thrown in
  45. 45. 43 Results • “Monolith Plus” architecture • Extract services that solve real problems, not imagined ones • Avoid SOA “tax”
  46. 46. 44 So my primary guideline would be don’t even consider microservices unless you have a system that’s too complex to manage as a monolith - Martin Fowler, “MicroservicePremium”
  47. 47. 45 Conclusions
  48. 48. 46 Conclusions 1. Our architecture is boring 2. How we keep it boring is interesting: 1. Start with what we know 2. Measure it live 3. Fix the slow
  49. 49. 47 Application • You can optimize for performance and get scale thrown in (almost for free) • Your monolith can scale further than you think • SOA is not the only way – Know your own problem space – Fix actual problems
  50. 50. 48 Questions? (We’re all about questions) Obligatory: • We’re hiring! stackexchange.com/work-here • Open source! stackexchange.github.io • Follow me! twitter.com/df07
  51. 51. 49
  52. 52. 50 Here Be Dragons (rejected slides)
  53. 53. 51 • Started with basic OutputCache (cache rendered HTML for a page) • ~4% cache hit rate Caching
  54. 54. 52 • Add in-memory & Redis caching Caching
  55. 55. 53 StackExchange.Redis • Wrote our own library for talking to Redis • Multiplexing operations over a single connection • Aware of primary / secondary instances – Can target reads at secondary slave
  56. 56. 54 StackExchange.Redis (opserver – https://github.com/opserver/opserver)
  57. 57. 55 Moonspeak (Localization)
  58. 58. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/stack- exchange
  • AjmalAshraf16

    Oct. 3, 2021
  • neb85

    Aug. 3, 2020
  • RajivChauhan2

    May. 25, 2020
  • HossamMousa3

    Apr. 17, 2020
  • kanenathan213

    Nov. 11, 2019
  • eng333

    Sep. 10, 2019
  • maemichiyuya

    Dec. 16, 2018
  • rnjailamba12

    Jul. 3, 2018
  • ConanChou

    Jun. 25, 2018
  • ChrisBecker21

    Apr. 20, 2018
  • vishwasm

    Apr. 17, 2018

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1LjPgnq. David Fullerton shares some of the things the Stack Exchange tech team have learned along the way while scaling one of the top sites in the world primarily through vertical scaling. Filmed at qconnewyork.com. David Fullerton is the developer-turned-manager responsible for all software development and system administration at Stack Exchange. David joined Stack Exchange, Inc. from Fog Creek Software, where he worked briefly as the developer lead for Stack Exchange 1.0 and as a developer on the FogBugz team. He has a B.A. in Computer Science from Dartmouth College.

Views

Total views

11,862

On Slideshare

0

From embeds

0

Number of embeds

828

Actions

Downloads

0

Shares

0

Comments

0

Likes

11

×