Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
@NYTDevs | developers.nytimes.com
@NYTDevs | developers.nytimes.com
Varnish: Linchpin of the
NYTimes.com Re-architecture
Adam E. Falk
Software Architect, We...
@NYTDevs | developers.nytimes.com
Who I Am
A software architect focusing on server
configuration and resiliency, with
side...
@NYTDevs | developers.nytimes.com
Who We Are
Photo credit: Tony Cenicola/The New York Times
@NYTDevs | developers.nytimes.com
Scope of this Presentation
Everything that follows pertains to the use of
Varnish to acc...
@NYTDevs | developers.nytimes.com
NYTimes.com: Size
15+ million page URLs (1851–present)
● Not all HTML; working on that
2...
@NYTDevs | developers.nytimes.com
NYTimes.com: Traffic
<www.nytimes.com> normal daily peak is
~75,000 requests/second – ju...
@NYTDevs | developers.nytimes.com
2013 Redesign of NYTimes.com
@NYTDevs | developers.nytimes.com
Mission Statement
“Leverage the latest technology in order to
improve the user experienc...
@NYTDevs | developers.nytimes.com
Improve the User Experience
Technical goals:
1. 25% improvement in browser load time, mi...
@NYTDevs | developers.nytimes.com
50% or better improvement in
● Time to first byte
● Time to paint
● Time to page ready
A...
@NYTDevs | developers.nytimes.com
Brave New World
@NYTDevs | developers.nytimes.com
Exception to the Rule
A complete code rewrite (almost). Why?
● < insert usual suspects h...
@NYTDevs | developers.nytimes.com
Never Let a Crisis Go To Waste
☒ (Test|Behavior) Driven Development
☒ Web performance wa...
@NYTDevs | developers.nytimes.com
Can We Cache Pages Now?
Yes, Virginia.
</summary>
@NYTDevs | developers.nytimes.com
Spotlights for You
VCL file modular organization
Cache refresh instead of purge
Varnish ...
@NYTDevs | developers.nytimes.com
Changing Horses in Midstream
Site functionality that must not break:
● redirects (mobile...
@NYTDevs | developers.nytimes.com
Best Practice (singular)
@NYTDevs | developers.nytimes.com
Easy Yet Powerful
@NYTDevs | developers.nytimes.com
Easy Yet Powerful
@NYTDevs | developers.nytimes.com
Easy Yet Powerful
@NYTDevs | developers.nytimes.com
Easy Yet Powerful
@NYTDevs | developers.nytimes.com
Greatest Thing Since Sliced Bread
☒ Single responsibility principle
☒ Code readability (...
@NYTDevs | developers.nytimes.com
Intermission
There are only two hard things in
Computer Science:
1. Cache invalidation
2...
@NYTDevs | developers.nytimes.com
Cache Invalidation
Purge is not good enough (in Varnish 3).
PURGE causes cache misses on...
@NYTDevs | developers.nytimes.com
NYT Homepage
● Must always be in Varnish cache.
● Every article linked to on the homepag...
@NYTDevs | developers.nytimes.com
But...
Some content changes frequently.
Latest version served in real-time after every
p...
@NYTDevs | developers.nytimes.com
Cache Rules Everything Around Me
CREAM: an API to re(set|build|prime) a single
cache ent...
@NYTDevs | developers.nytimes.com
req.hash_always_miss = true
CREAM requests the just-updated article to
every Varnish ser...
@NYTDevs | developers.nytimes.com
Where We Are Today: Software
~2,300 lines of VCL code
● Minimum of inline C
10 VMODs
● s...
@NYTDevs | developers.nytimes.com
Where We Are Today: Traffic
Of the ~4,000 page requests/second to
<www.nytimes.com>:
● ~...
@NYTDevs | developers.nytimes.com
Where We Are Today: Performance
Load test: ~3,000 requests/second/server with
current co...
@NYTDevs | developers.nytimes.com
8 Servers? Why?!
Because:
● Biggest spike ever was 10x (2012 Election Night)
● 2 hypervi...
@NYTDevs | developers.nytimes.com
Next Steps for Us
1. Install Varnish Cache Plus 4
2. Utilize the Varnish Plus tools for ...
@NYTDevs | developers.nytimes.com
Thank You
Adam E. Falk
falkae@nytimes.com
@xenograg
adamfalk.com xenograg.com
We’re hiring
nytimes.com/careers
@NYTDevs | #timesopen | developers.nytimes.com
Upcoming SlideShare
Loading in …5
×

NY Times: so news doesn't break your server

310 views

Published on

NY Times talk at Varnish Summit, New York 2015

Published in: Internet
  • Be the first to comment

  • Be the first to like this

NY Times: so news doesn't break your server

  1. 1. @NYTDevs | developers.nytimes.com
  2. 2. @NYTDevs | developers.nytimes.com Varnish: Linchpin of the NYTimes.com Re-architecture Adam E. Falk Software Architect, Web Products
  3. 3. @NYTDevs | developers.nytimes.com Who I Am A software architect focusing on server configuration and resiliency, with sidelines in DevOps, release engineering, and testing. Started as a LAMP developer but has always been a generalist interested in all aspects of the data center.
  4. 4. @NYTDevs | developers.nytimes.com Who We Are Photo credit: Tony Cenicola/The New York Times
  5. 5. @NYTDevs | developers.nytimes.com Scope of this Presentation Everything that follows pertains to the use of Varnish to accelerate serving content on the <www.nytimes.com> hostname, only. There are several other Varnish clusters at NYTimes.com.
  6. 6. @NYTDevs | developers.nytimes.com NYTimes.com: Size 15+ million page URLs (1851–present) ● Not all HTML; working on that 200+ new page URLs created each day Millions more image URLs
  7. 7. @NYTDevs | developers.nytimes.com NYTimes.com: Traffic <www.nytimes.com> normal daily peak is ~75,000 requests/second – just this hostname. ● primarily APIs ● HTML traffic is ~4,000 req/sec Traffic spikes up to 4x during a breaking news event R.I.P. Leonard Nimoy
  8. 8. @NYTDevs | developers.nytimes.com 2013 Redesign of NYTimes.com
  9. 9. @NYTDevs | developers.nytimes.com Mission Statement “Leverage the latest technology in order to improve the user experience, enhance our journalism, and provide a more effective environment for our advertisers.” Project document
  10. 10. @NYTDevs | developers.nytimes.com Improve the User Experience Technical goals: 1. 25% improvement in browser load time, minimum. 2. ... Sounds like a job for page caching!
  11. 11. @NYTDevs | developers.nytimes.com 50% or better improvement in ● Time to first byte ● Time to paint ● Time to page ready Achievement Unlocked
  12. 12. @NYTDevs | developers.nytimes.com Brave New World
  13. 13. @NYTDevs | developers.nytimes.com Exception to the Rule A complete code rewrite (almost). Why? ● < insert usual suspects here > ● Deeply embedded server-side personalization (includes ads) Output was simply uncacheable.
  14. 14. @NYTDevs | developers.nytimes.com Never Let a Crisis Go To Waste ☒ (Test|Behavior) Driven Development ☒ Web performance was core from Day 0 ☒ Async wherever, whenever ☒ New APIs ☒ CSS: LESS (then), SASS (now)
  15. 15. @NYTDevs | developers.nytimes.com Can We Cache Pages Now? Yes, Virginia. </summary>
  16. 16. @NYTDevs | developers.nytimes.com Spotlights for You VCL file modular organization Cache refresh instead of purge Varnish cluster today
  17. 17. @NYTDevs | developers.nytimes.com Changing Horses in Midstream Site functionality that must not break: ● redirects (mobile, registration, et. al.) ● user tracking ● web crawler detection
  18. 18. @NYTDevs | developers.nytimes.com Best Practice (singular)
  19. 19. @NYTDevs | developers.nytimes.com Easy Yet Powerful
  20. 20. @NYTDevs | developers.nytimes.com Easy Yet Powerful
  21. 21. @NYTDevs | developers.nytimes.com Easy Yet Powerful
  22. 22. @NYTDevs | developers.nytimes.com Easy Yet Powerful
  23. 23. @NYTDevs | developers.nytimes.com Greatest Thing Since Sliced Bread ☒ Single responsibility principle ☒ Code readability (and understanding!) ☒ Time spent troubleshooting ☒ Coding standards
  24. 24. @NYTDevs | developers.nytimes.com Intermission There are only two hard things in Computer Science: 1. Cache invalidation 2. Naming things 3. Off-by-one errors http://martinfowler.com/bliki/TwoHardThings.html
  25. 25. @NYTDevs | developers.nytimes.com Cache Invalidation Purge is not good enough (in Varnish 3). PURGE causes cache misses on the highest- traffic content. Needed cache re(set|build|prime).
  26. 26. @NYTDevs | developers.nytimes.com NYT Homepage ● Must always be in Varnish cache. ● Every article linked to on the homepage should already be in Varnish cache. No cache misses = long TTL.
  27. 27. @NYTDevs | developers.nytimes.com But... Some content changes frequently. Latest version served in real-time after every publish action. Short TTL = more cache misses. PURGE = more cache misses.
  28. 28. @NYTDevs | developers.nytimes.com Cache Rules Everything Around Me CREAM: an API to re(set|build|prime) a single cache entry. Publish event calls API synchronously.
  29. 29. @NYTDevs | developers.nytimes.com req.hash_always_miss = true CREAM requests the just-updated article to every Varnish server, in parallel.
  30. 30. @NYTDevs | developers.nytimes.com Where We Are Today: Software ~2,300 lines of VCL code ● Minimum of inline C 10 VMODs ● std, utils, crashhandler, wurfl, boltsort, queryfilter ● 4 custom
  31. 31. @NYTDevs | developers.nytimes.com Where We Are Today: Traffic Of the ~4,000 page requests/second to <www.nytimes.com>: ● ~1,500 now served by Varnish ● ~91% cache hit rate (down from ~96%)
  32. 32. @NYTDevs | developers.nytimes.com Where We Are Today: Performance Load test: ~3,000 requests/second/server with current configuration We could handle a 4x spike with 2 servers We run 8 servers per data center
  33. 33. @NYTDevs | developers.nytimes.com 8 Servers? Why?! Because: ● Biggest spike ever was 10x (2012 Election Night) ● 2 hypervisors => even number of server instances ● Takes too long for us to dynamically provision ● We can afford to stay over-provisioned Yes, this causes extra backend network traffic. Scaled out for resilience, scaling up for performance.
  34. 34. @NYTDevs | developers.nytimes.com Next Steps for Us 1. Install Varnish Cache Plus 4 2. Utilize the Varnish Plus tools for monitoring. 3. Replace CREAM with VHA
  35. 35. @NYTDevs | developers.nytimes.com Thank You Adam E. Falk falkae@nytimes.com @xenograg adamfalk.com xenograg.com
  36. 36. We’re hiring nytimes.com/careers @NYTDevs | #timesopen | developers.nytimes.com

×