This document summarizes the scaling challenges at Showyou and the approaches taken to address them. It discusses moving from MySQL to the more scalable Riak and Redis for storage, taking a microservices approach to processing with many small discrete services, and building a custom monitoring system called UState to aggregate states across services and infrastructure. The overall approach emphasizes robustness, distribution, eventual consistency, and comprehensive monitoring.

1. Introduction Storage Processing Monitoring Review
Scaling at Showyou
Operations
September 26, 2011

2. Introduction Storage Processing Monitoring Review
I’m Kyle Kingsbury
Handle aphyr
Code http://github.com/aphyr
Email kyle@remixation.com
Focus Backend, API, ops

3. Introduction Storage Processing Monitoring Review
What the hell is Showyou?

5. Introduction Storage Processing Monitoring Review
Nontrivial complexity

6. Introduction Storage Processing Monitoring Review
Challenges
ˆ Scanning social networks
ˆ Feeds
ˆ Search
ˆ Trends
ˆ Responsive client experience

7. Introduction Storage Processing Monitoring Review
Challenges
ˆ Scanning social networks
ˆ Feeds
ˆ Search
ˆ Trends
ˆ Responsive client experience
ˆ Everything fails all the time

9. Introduction Storage Processing Monitoring Review
Storage

11. Introduction Storage Processing Monitoring Review
We left MySQL

12. Introduction Storage Processing Monitoring Review
We left MySQL
ˆ Changing the schema requires downtime
ˆ Crashes
ˆ Master-slave lag
ˆ Slow restarts
ˆ Node replacements difficult
ˆ Fully normalized queries complex, slow

13. Introduction Storage Processing Monitoring Review
MySQL does scale
But there are tradeoffs

14. Introduction Storage Processing Monitoring Review
Riak
ˆ Key/value store
ˆ Homogenous
ˆ Scales linearly with nodes
ˆ Excellent durability/recoverability
ˆ Eventually consistent

15. Introduction Storage Processing Monitoring Review
We use Riak as our durable datastore
ˆ Users, feeds, videos, etc
ˆ Highly denormalized
ˆ Limited MR queries (feeds, etc)
ˆ Latency-bounded MR jobs are Erlang
ˆ Hot-deployable
ˆ Extensive use of conflict resolution
ˆ Made possible by Risky

16. Introduction Storage Processing Monitoring Review
Riak at Showyou
ˆ 51 million keys (153 M replicated)
ˆ 100 GB of data (300 GB replicated)
ˆ 260 gets/sec (baseline)
ˆ 75 puts/sec (baseline)
ˆ Capable of over 3000 ops/sec

17. Introduction Storage Processing Monitoring Review
SSDs are amazing
WD 7200RPM Micron RealSSD P300
ˆ 100 ops/sec ˆ 1000+ ops/sec
ˆ 95%: 100-300ms ˆ 95%: 3-5ms

18. Introduction Storage Processing Monitoring Review
When Riak fails,
ˆ Another node takes up the slack
ˆ Clients connected to that node reconnect to others
ˆ Typically no service interruption
ˆ However, latencies may rise
ˆ Especially for MR jobs

19. Introduction Storage Processing Monitoring Review
Riak has downsides
ˆ Difficult to debug
ˆ Membership changes are dangerous
ˆ Significantly slower than MySQL
ˆ (Bitcask) All keys must fit in memory
ˆ Mapreduce is only appropriate for known keys
ˆ List-keys can take down your cluster
Long story short: it’s only a KV store

20. Introduction Storage Processing Monitoring Review
+Redis

21. Introduction Storage Processing Monitoring Review
We use Redis for fast, temporary state
ˆ List of users
ˆ List of videos
ˆ Counters
ˆ Queues
Incredibly fast, excellent primitives

22. Introduction Storage Processing Monitoring Review
When Redis fails,
ˆ Daemons using those indexes pause
ˆ Frontend service continues
ˆ Bitcask scanners and incremental updaters repair
any lost data

23. Introduction Storage Processing Monitoring Review
When Redis fails,
ˆ Daemons using those indexes pause
ˆ Frontend service continues
ˆ Bitcask scanners and incremental updaters repair
any lost data
Eventually consistent.

24. Introduction Storage Processing Monitoring Review
We also use SOLR extensively
ˆ Supplements Riak
ˆ Complex indices
ˆ Full-text search
ˆ Analytics
More on that later. . .

25. Introduction Storage Processing Monitoring Review
Processing

27. Introduction Storage Processing Monitoring Review
Do one thing well
Lots of small processes handling well-defined tasks
ˆ Easier to debug
ˆ Easier to test
ˆ Simplifies parallelism
ˆ Simplifies error handling
ˆ Less likely to cause total system failure

28. Introduction Storage Processing Monitoring Review
Minimize Shared State
ˆ Vector clocks for concurrent modification
ˆ Queues for message passing
ˆ Riak for durable storage
ˆ Redis for fast synchronous state

29. Introduction Storage Processing Monitoring Review
Crash by Default
ˆ Someone else will take your work
ˆ Repair constantly
ˆ Assume everybody is out to kill you

30. Introduction Storage Processing Monitoring Review
Distribute
ˆ Multiple threads, processes, hosts
ˆ Failover IPs with Heartbeat
ˆ Rolling restarts mean frequent deploys and nobody
notices
ˆ Losing a node is no big deal
ˆ Scaling out is easy

32. Introduction Storage Processing Monitoring Review
Monitoring

34. Introduction Storage Processing Monitoring Review
UState: A state aggregator

35. Introduction Storage Processing Monitoring Review
Receive states over protobufs
Host backend1.showyou.com
Service feed merger rate
Time unix epoch seconds
State ok
Metric 12.5
Description 12.5 feed items/sec

36. Introduction Storage Processing Monitoring Review
Query states
ˆ state = "warning" or state = "critical"
ˆ service =∼ "api %" and host != null

37. Introduction Storage Processing Monitoring Review
ˆ Combine states together (sum, average, . . . )
ˆ Send email on changes
ˆ Forward to another UState server
ˆ Forward to Graphite
ˆ Dashboard

40. Introduction Storage Processing Monitoring Review
Understand application behavior

42. Introduction Storage Processing Monitoring Review
When can we. . . ?

44. Introduction Storage Processing Monitoring Review
It’s 23:15 PST.

45. Introduction Storage Processing Monitoring Review
It’s 23:15 PST.
Do you know where YOUR database is?

53. Introduction Storage Processing Monitoring Review
http://github.com/aphyr/ustate

54. Introduction Storage Processing Monitoring Review
Recap
ˆ Robust, discrete components
ˆ Highly distributed
ˆ Message passing
ˆ Eventual consistency
ˆ Comprehensive monitoring

55. Introduction Storage Processing Monitoring Review
Thanks!
ˆ Basho (esp. Pharkmillups!)
ˆ Formspring
ˆ Bump