The document provides an overview of Tumblr, a platform founded in 2007, which hosts around 200 million blogs and over 83 billion posts. It details the technical stack used by Tumblr, including both frontend and backend technologies, and elaborates on their processes for deploying features, error logging, and the implementation of feature flagging for controlled A/B testing. Chris Miller, the product engineering manager, discusses the importance of quick iteration, thorough monitoring, and how they handle error logging to maintain service reliability.

2. Frontend at Scale
the Tumblr story

3. What is Tumblr?
→ Platform for you to express yourself
→ ~200 million blogs
→ 83+ billion posts
→ HQ in NYC
→ Founded in 2007
→ 100+ engineers

4. What is Tumblr?
→ Three ways to surface
content:
→ The dashboard

5. What is Tumblr?
→ Three ways to surface
content:
→ The dashboard
→ Search

6. What is Tumblr?
→ Three ways to surface
content:
→ The dashboard
→ Search
→ Blog network
!
(Example: http://16-bitch.tumblr.com/)

7. Who am I?
→ Chris Miller
→ Product Engineering Manager
→ Content Consumption (a.k.a., The Dashboard)

8. Our stack
→ Frontend
→ Backbone (+ lodash, underscore, etc.)
→ jQuery (+ some plugins)
→ SASS (+ Bourbon)
→ a bit of VelocityJS
→ Gulp for build

9. Our stack
→ Backend
→ PHP application layer
→ Some specialized services (Scala, C, etc.)
→ Data: MySQL, Redis, memcache, HDFS

10. How does it work?
→ 1000’s of servers
→ Deploy dozens of times per day
→ Monitor and measure everything
→ Hadoop
→ OpenTSDB (backed by HBase)

11. Our process
→ Teams are small
→ Iterate quickly
→ Release early and often, usually to % of users
→ 2 code review “ok’s” required for all Pull Requests

12. Feature Flagging

13. Feature Flagging
What is it?
→ Segregate your users to certain features
→ Control who sees what (and when)

14. Feature Flagging
Implementation
→ Server-side feature flagging
→ Client-side feature flagging

15. Feature Flagging
Usage
→ Provides
→ A/B testing
→ Run beta code alongside production code
→ Kill switch

16. Feature Flagging
A/B Testing
→ Injected recommendations
→ A/B(/*) testing of
positioning
→ Which position is the
best? Why?

17. Feature Flagging
A/B Test Results
→ Injected recommendations
→ A/B(/*) testing of
positioning
→ Which position is the
best? Why?
Position 2
Position 3
Position 4
Position 5
Position 6
Position 7
Position 8
Position 9

18. Feature Flagging
Ramping & Kill Switch
→ Ramping new features
→ Deploy to only “admin” (staff)
→ …then 1% of users… then 5%… 10%… 25%…
→ Kill switch
→ Completely turn off a feature that’s breaking the site… poof

19. Feature Flagging
Use Carefully
→ Feature flagging certain functionality can give a mixed
experience
→ Can cause user confusion:
→ “Why does my mom see this and I don’t?”
— Confused teenager
→ Easy to build complex dependencies — don’t

20. Error Logging

21. Error Logging
Launching Features
→ New features usually have bugs
→ (Well, not my code)
→ (just kidding)

22. Error Logging
Error Logging
→ New features usually have bugs
→ Server-side errors, easy to find

23. Error Logging
Error Logging
→ New features usually have bugs
→ Client-side errors, also easy to find…
→ …on my browser

24. Error Logging
Error Logging
→ New features usually have bugs
→ Client-side errors, not easy to find on your browser
→ …until recently

25. Error Logging
Capture Errors
→ We built: exceptions.js
→ Really, it’s just: window.onerror

26. Error Logging
Capture Errors
→ Build dependency-free
→ Build to be defensive

27. Error Logging
Capture Errors
→ What you do with the logs doesn’t matter; it’s how you use it
→ We log errors to Scribe…
→ …throw them into Hadoop
→ …and count frequency with OpenTSDB

28. Error Logging
Error Data
→ With Hive, we can query Hadoop:
→ With this, I can see we log around 1.4 million errors per day

29. Error Logging
Error Data
→ With OpenTSDB we can plot the frequency of logs

30. Error Logging
We Love Graphs
→ We made pretty graphs with OpenTSDB and graph everything

31. Getting it Right
→ Sometimes we find errors before our users do.
→ Sometimes.
→ And it makes us feel good.

32. Getting it Right
→ So we dance.

33. Thank You
Email - cmiller@tumblr.com
Follow me - ee99ee.com