Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Economics of Scale Tyler Treat

WorkivaPromises and Perils of Going Distributed
September 19, 2015
About The Speaker
• Backend engineer at Workiva
• Messaging platform tech lead
• Distributed systems
• bravenewgeek.com @t...
About The Talk
• Why distributed systems?
• Case study
• Advantages/Disadvantages
• Strategies for scaling and resilience ...
What does it mean to “scale” a system?
Scale Up vs. Scale Out
❖ Add resources to a node

❖ Increases node capacity, load
is unaffected

❖ System complexity unaffec...
Okay, cool—but what does this actually mean?
Let’s look at a real-world example…
How does Twitter work?
Just write tweets to a big database table.
Getting a timeline is a simple join query.
But…this is crazy expensive.
Joins are sloooooooooow.
Prior to 5.5, MySQL used table-level locking. Now
it uses row-level locking. Either way,

lock contention everywhere.
As the table grows larger, lock contention on
indexes becomes worse too.
And UPDATES get higher priority than SELECTS.
So now what?
Distribute!
Specifically, shard.
Partition tweets into different databases using
some consistent hash scheme

(put a hash ring on it).
This alleviates lock contention and improves
throughput…



but fetching timelines is still extremely costly
(now scatter-...
Observation: Twitter is a consumption mechanism
more than an ingestion one…
i.e. cheap reads > cheap writes
Move tweet processing to the write path

rather than read path.
Ingestion/Fan-Out Process
1. Tweet comes in
2. Query the social graph service for followers
3. Iterate through each follow...
Ingestion/Fan-Out Process
• Lots of processing on ingest, no computation on reads
• Redis stores timelines in memory—very ...
Key Takeaway: think about your access patterns
and design accordingly.


Optimize for the critical path.
Let’s Recap…
• Advantages of single database system:
• Simple!
• Data and invariants are consistent (ACID transactions)
• ...
Going distributed solved the problem,

but at what cost?
(hint: your sanity)
Distributed systems are all about trade-offs.
By choosing availability, we give up consistency.
This problem happens all the time on Twitter.



For example, you tweet, someone else replies, and
I see the reply before ...
Can we solve this problem?
Sure, just coordinate things before proceeding…
“Have you seen this tweet? Okay, good.”
“Have you seen this tweet? Okay, g...
Sooo what do you do when Justin Bieber tweets to
his 67 million followers?
Coordinating for consistency is expensive

when data is distributed

because processes

can’t make progress independently.
Source: Peter Bailis, 2015 https://speakerdeck.com/pbailis/silence-is-golden-coordination-avoiding-systems-design
Key Takeaway: strong consistency is slow and
distributed coordination is expensive (in terms of
latency and throughput).
Sharing mutable data at large scale is difficult.
If we don’t distribute, we risk scale problems.
Let’s say we want to count the number of times a
tweet is retweeted.
“Get, add 1, and put”
transaction will not

scale.
If we do distribute, we risk consistency problems.
What do we do when our system is partitioned?
If we allow writes on both sides of the partition,
how do we resolve conflicts when the partition
heals?
Distributed systems are hard!
But lots of good research going on to solve these
problems…
CRDTs
Lasp
SyncFree
RAMP transactions
etc.
Twitter has 316 million monthly active users.
Facebook has 1.49 billion monthly active users.
Netflix has 62.3 million stre...
How do you build resilient systems at this scale?
Embrace failure.
Provide partial availability.
If an overloaded service is not essential to

core business, fail fast to prevent availability or
latency problems upstrea...
It’s better to fail predictably than fail in
unexpected ways.
Use backpressure to reduce load.
Flow-Control Mechanisms
• Rate limit
• Bound queues/buffers
• Backpressure - drop messages on the floor
• Increment stat co...
Bounding resource utilization and failing fast
helps maintain predictable performance and
impedes cascading failures.
Going distributed means more wire time.

How do you improve performance?
Cache everything.
“There are only two hard things in computer
science: cache invalidation and naming
things.”
Embrace asynchrony.
Distributed systems are not just about workload
scale, they’re about organizational scale.
In 2010, Workiva released a product to streamline
financial reporting.
A specific solution to solve a very specific
problem, originally built by a few dozen
engineers.
Fast forward to today:
a couple hundred engineers,
more users, more markets, more solutions.
How do you ramp up new products quickly?
You stop thinking in terms of products and start
thinking in terms of platform.
From Product to Platform
At this point, going distributed is all but necessary.
Service-Oriented Architecture allows us to
independently build, deploy, and scale discrete
parts of the platform.
Loosely coupled services let us tolerate failure.
And things fail constantly.
Shit happens — network partitions, hardware
failure, GC pauses, latency, dropped packets…
Build resilient systems.
–Ken Arnold
“You have to design distributed systems with
the expectation of failure.”
Design for failure.
Consider the trade-off between
consistency and availability.
Most important?
Don’t distribute until you have
a reason to!
Scale up until you have to

scale out.
–Paul Barham
“You can have a second computer once you’ve
shown you know how to use the first one.”
And when you do distribute,

don’t go overboard.
Walk before you run.
Remember, when it comes to
distributed systems…

for every promise there’s a peril.
Thanks!
@tyler_treat

github.com/tylertreat

bravenewgeek.com
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
Upcoming SlideShare
Loading in …5
×

The Economics of Scale: Promises and Perils of Going Distributed

986 views

Published on

What does it take to scale a system? We'll learn how going distributed can pay dividends in areas like availability and fault tolerance by examining a real-world case study. However, we will also look at the inherent pitfalls. When it comes to distributed systems, for every promise there is a peril.

Published in: Software
  • Be the first to comment

The Economics of Scale: Promises and Perils of Going Distributed

  1. 1. The Economics of Scale Tyler Treat WorkivaPromises and Perils of Going Distributed September 19, 2015
  2. 2. About The Speaker • Backend engineer at Workiva • Messaging platform tech lead • Distributed systems • bravenewgeek.com @tyler_treat tyler.treat@workiva.com
  3. 3. About The Talk • Why distributed systems? • Case study • Advantages/Disadvantages • Strategies for scaling and resilience patterns • Scaling Workiva
  4. 4. What does it mean to “scale” a system?
  5. 5. Scale Up vs. Scale Out ❖ Add resources to a node ❖ Increases node capacity, load is unaffected ❖ System complexity unaffected Vertical Scaling ❖ Add nodes to a cluster ❖ Decreases load, capacity is unaffected ❖ Availability and throughput w/ increased complexity Horizontal Scaling
  6. 6. Okay, cool—but what does this actually mean?
  7. 7. Let’s look at a real-world example…
  8. 8. How does Twitter work?
  9. 9. Just write tweets to a big database table.
  10. 10. Getting a timeline is a simple join query.
  11. 11. But…this is crazy expensive.
  12. 12. Joins are sloooooooooow.
  13. 13. Prior to 5.5, MySQL used table-level locking. Now it uses row-level locking. Either way,
 lock contention everywhere.
  14. 14. As the table grows larger, lock contention on indexes becomes worse too.
  15. 15. And UPDATES get higher priority than SELECTS.
  16. 16. So now what?
  17. 17. Distribute!
  18. 18. Specifically, shard.
  19. 19. Partition tweets into different databases using some consistent hash scheme
 (put a hash ring on it).
  20. 20. This alleviates lock contention and improves throughput…
 
 but fetching timelines is still extremely costly (now scatter-gather query across multiple DBs).
  21. 21. Observation: Twitter is a consumption mechanism more than an ingestion one… i.e. cheap reads > cheap writes
  22. 22. Move tweet processing to the write path
 rather than read path.
  23. 23. Ingestion/Fan-Out Process 1. Tweet comes in 2. Query the social graph service for followers 3. Iterate through each follower and insert tweet ID into their timeline (stored in Redis) 4. Store tweet on disk (MySQL)
  24. 24. Ingestion/Fan-Out Process • Lots of processing on ingest, no computation on reads • Redis stores timelines in memory—very fast • Fetching timeline involves no queries—get timeline from Redis cache and rehydrate with multi-get on IDs • If timeline falls out of cache, reconstitute from disk • O(n) on writes, O(1) on reads • http://www.infoq.com/presentations/Twitter-Timeline-Scalability
  25. 25. Key Takeaway: think about your access patterns and design accordingly. 
 Optimize for the critical path.
  26. 26. Let’s Recap… • Advantages of single database system: • Simple! • Data and invariants are consistent (ACID transactions) • Disadvantages of single database system: • Slow • Doesn’t scale • Single point of failure
  27. 27. Going distributed solved the problem,
 but at what cost? (hint: your sanity)
  28. 28. Distributed systems are all about trade-offs.
  29. 29. By choosing availability, we give up consistency.
  30. 30. This problem happens all the time on Twitter.
 
 For example, you tweet, someone else replies, and I see the reply before the original tweet.
  31. 31. Can we solve this problem?
  32. 32. Sure, just coordinate things before proceeding… “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.”
  33. 33. Sooo what do you do when Justin Bieber tweets to his 67 million followers?
  34. 34. Coordinating for consistency is expensive
 when data is distributed
 because processes
 can’t make progress independently.
  35. 35. Source: Peter Bailis, 2015 https://speakerdeck.com/pbailis/silence-is-golden-coordination-avoiding-systems-design
  36. 36. Key Takeaway: strong consistency is slow and distributed coordination is expensive (in terms of latency and throughput).
  37. 37. Sharing mutable data at large scale is difficult.
  38. 38. If we don’t distribute, we risk scale problems.
  39. 39. Let’s say we want to count the number of times a tweet is retweeted.
  40. 40. “Get, add 1, and put” transaction will not
 scale.
  41. 41. If we do distribute, we risk consistency problems.
  42. 42. What do we do when our system is partitioned?
  43. 43. If we allow writes on both sides of the partition, how do we resolve conflicts when the partition heals?
  44. 44. Distributed systems are hard!
  45. 45. But lots of good research going on to solve these problems… CRDTs Lasp SyncFree RAMP transactions etc.
  46. 46. Twitter has 316 million monthly active users. Facebook has 1.49 billion monthly active users. Netflix has 62.3 million streaming subscribers.
  47. 47. How do you build resilient systems at this scale?
  48. 48. Embrace failure.
  49. 49. Provide partial availability.
  50. 50. If an overloaded service is not essential to
 core business, fail fast to prevent availability or latency problems upstream.
  51. 51. It’s better to fail predictably than fail in unexpected ways.
  52. 52. Use backpressure to reduce load.
  53. 53. Flow-Control Mechanisms • Rate limit • Bound queues/buffers • Backpressure - drop messages on the floor • Increment stat counters for monitoring/alerting • Exponential back-off • Use application-level acks for critical transactions
  54. 54. Bounding resource utilization and failing fast helps maintain predictable performance and impedes cascading failures.
  55. 55. Going distributed means more wire time.
 How do you improve performance?
  56. 56. Cache everything.
  57. 57. “There are only two hard things in computer science: cache invalidation and naming things.”
  58. 58. Embrace asynchrony.
  59. 59. Distributed systems are not just about workload scale, they’re about organizational scale.
  60. 60. In 2010, Workiva released a product to streamline financial reporting.
  61. 61. A specific solution to solve a very specific problem, originally built by a few dozen engineers.
  62. 62. Fast forward to today: a couple hundred engineers, more users, more markets, more solutions.
  63. 63. How do you ramp up new products quickly?
  64. 64. You stop thinking in terms of products and start thinking in terms of platform.
  65. 65. From Product to Platform
  66. 66. At this point, going distributed is all but necessary.
  67. 67. Service-Oriented Architecture allows us to independently build, deploy, and scale discrete parts of the platform.
  68. 68. Loosely coupled services let us tolerate failure. And things fail constantly.
  69. 69. Shit happens — network partitions, hardware failure, GC pauses, latency, dropped packets…
  70. 70. Build resilient systems.
  71. 71. –Ken Arnold “You have to design distributed systems with the expectation of failure.”
  72. 72. Design for failure.
  73. 73. Consider the trade-off between consistency and availability.
  74. 74. Most important?
  75. 75. Don’t distribute until you have a reason to!
  76. 76. Scale up until you have to
 scale out.
  77. 77. –Paul Barham “You can have a second computer once you’ve shown you know how to use the first one.”
  78. 78. And when you do distribute,
 don’t go overboard. Walk before you run.
  79. 79. Remember, when it comes to distributed systems…
 for every promise there’s a peril.
  80. 80. Thanks! @tyler_treat github.com/tylertreat
 bravenewgeek.com

×